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Chapter I 



INTRODUCTION 



Assessment has become one of the leading topics in education 
in the 1980's. Calls from national and state leaders for reform 
and quality improvement of higher education began in the early 
part of the decade and have led inevitably to the need for more 
assessment - to see if reforms are working and improvements are 
being achieved. Current state and national interest in 
evaluation is motivated by the need to have better measures of 
how well our large, important, and expensive higher education 
systems are working . The effort to measure results is probably 
just beginning. This report is an effort to determine how 
evaluation experiences oi' the recent decades can show us where 
these efforts are leading state systems and the agencies which 
attempt to lead the systems. 

Assessment is not new in education; in one form or another 
it is as old as education itself. American students are 
evaluated in each course, and credentialling and licensing exams 
occur at the end of many programs of study. Institutions are 
accredited and reaccredited through an evaluation process. 
Faculty are evaluated for tenure, promotion and merit pay 
increases. Research projects undergo peer review. Studies of 
whole subject areas such as medicine or engineering also occur 
from time to time. And, of course, appraisals are made of state 
systems of education. 

What is new are systematic and comprehensive state efforts 
to measure the effectiveness and the outcomes of education 
institutions and systems on a regular basis . Although some state 
efforts predated it, the federal National Institutes of Education 
(NIE) Report, A Nation at Risk (1983), is generally credited with 
bringing the issue of (elementary/secondary) educational quality 
dramatically to the public's attention. The next year, NIE 
sponsored another study. Involvement in Learning , which along 
with several other reports raised the issue of quality at the 
higher education level . Those reports downplayed the federal 
role in education and pointed to the states, localities and 
higher education institutions as the major actors in educational 
reform. State leadership by governors and legislators was a 
major force pushing for more assessment. A recent report from 
the governors. Time for Results (1986), spells out the importance 
of ongoing assessment to stimulate as well as measure 
improvement . 

Evaluations of state higher education systems, and of the 
statewide boards and agencies that govern them, are varied in 
scope, method, and timing. This is because statewide planning 
and coordinating agencies are a recent pheno^nenon on the 
educational scene, and they vary greatly from state to state. 



Some are governing boards for all public institutions, some are 

coordinating agencies with limited regulatory powers, and others 

are planning and advisory groups. There is ntill uncertainty in * 

many states on what these state boards, commissions and agencies 

should be doing and how they should operate. As a result, the 

best evaluation model for these agencies has been elusive. 

The uncertainty about evaluation of the state systems is 
even greater. Higher education systems are so diverise, the ^ 
cause-and-ef f ect relationships so tenuous, that it is extremely 
difficult to say that a given factor should be changed in the 
hopes of achieving a different result. 

Nevertheless, a great deal of state evaluating of one kind 
or another is occurring, and this study will try to "assess the ^ 
assessments'* — to see whether certain processes seem more 
effective than others, and hew, if at all, they may fit together. 
Seven states were selected because of the variety of evaluations 
they had comp''eted during the last decade. The evaluations were 
examined thr interviews document reviews, and in other ways 

reported . later . The intent is not to look at detailed ^ 
institutional level evaluation processes, though it will be seen 
that such activity is critical to the success of state level 
assessments . 



The Empirical Basis for this Study • • 

Because more was needed than mere reading and review of 
various state evaluation documents, field visits v/ere made to the 
seven subject states. With no thought of deriving a ''scientific 
sample," the seven states were chosen ■'^o represent not only a 
variety of modes of evaluation but also different forms of 9 
statewide coordination and different regions of the country. 

Table One displays the seven states visited, their mode of 
state coordination and the evaluations relevant to thi?3 research. 
It will be seen that three of the seven have consolidated 
governing boards, three have coordinating boards, and one had an ^ 
advisory board at the time of the study. 



Consolidated Governing Board ^oordinat ing Board Advisory Board 

Florida Alabama Washington • 

Idaho Colorado 

West Virginia South Carolina 



Table Two represents the different modes of evaluations used 
by the states. The list of studies will reveal seven done by ^ 
blue ribbon commissions, eight by out-of-state consultants, two 
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TABLE ONE 



State 



Type of 
State Board 



Washington 



Advisory 
(Through 1986) 
Regulatory 
Coordinating 



since 



Relevant Evaluations 



!• ToTmrd A New Beginning! Balancing Local Control with 
State Coordination and Governance (Bowen and Glenny) , 
1984. 

2. Sunset Program and Fiscal Review of Council on Post- 
secondary Education , 1984. Report by the Legislative 
Budget Bureau 

3. Citizens* Conunittee with representation from both 
the House and Senate. Jan. '85 report The Paramount 
Duty 



Alabama 



Regulatory Coordinating 1. 



2. 



1st (1975), 2nd (1979) 3rd (1983) and 4th (1987) 
Quadrennial Evaluations of ACHE 
1983 Report of Council of Twenty One. Challenge- 
Obligation-Opportunity, The Imperative for Excellence 
in Higher Education 



Colorado 



Regulatory Coordinating 



1* Sunset Review - Committee on the Colorado Com- 
mission on Higher Education, June 1982. 

2. Report of the Higher Education Committee, January 
1985 



South Carolina Regulatory Coordinating 



1. Legislative Audit Council Report, 1978 

2. South C;arolina Master Plan for H.E. t 1980 

3. AVA Report to the Commission, Feb. 1986, Higher 
Education in S.C. : An Agenda for the Future 

4. T!he Cutting Edge: Higher Education's Initiatives for 
Excellence . Report of the South Carolina Commission 
on Higher Education 



Florida 



Consolidated Governing 



1. AED study A Call to Action , 1979 

2. Report and Recommendations of the Joint Legislative- 
Executive Commission on Postsecondary Education, 1980 

3. M aster Plan for Florida Postsecondary Education , 1982 

4. Regents Evaluative Criteria on the Delivery System, 
1983 and follow-ups, 1984-85 

5. A Path to Excellence in Public Higher Education in 
Florida, Regents* Study Commission on Funding for 
Excellence, 1985 
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TABLE ONE (Continued) 

State Type of Relevant Evaluations 

State Board 



West Virginia Consolidated Governing !• Support > Performance, and Protection of Hif^her 

Education in West Virginia ♦ Consultant (AED) study, 
1979 

2, Performance Audit - Sunset Review, 1984 

3, Agenda for Action - Master Plan, 1985-90 

4, Higher Education Study Commission Report, 1987 



Idaho Consolidated Governing !• The Future of Higher Education in Idaho , 1933 (ECS) 

2. Higher Education in Idaho: A Plan for the Future . 
(lACI), 1983 

3* Northwest Regional Report, 1984 ^ 

4, Strategic Plan, 1986, Directions for Higher Education 
Idaho: The Mext Five Years , 

5, Ad hoc Committee on H,E, (joint legislative interim 
committees) to examine state governance and 
community colleges, 1985 
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STATE HIGH2H 
EDUCAirOH SCA2D 



HLUS EI330N 
COKHISSIOS 



CU? OF STATS 
CONSULTANTS 



SUNSET 
Lf.GISLATIVB AUDIT 



2EFCH? 



Vashington, 1985 
The ParagQunt Duty 
Washington State 
Teaporary Coaaittes on 
Educational Policies, 



Structure, i Kanagezent and Sovenj 



Vashington, 193^ Vashington, 19S4 

T oward a Hew Beginning: Sunset Progras ani 

Balancing Local Control Fiscal Review 

with State Coordination Council on Pcst- 

and Governance (Gienny secondary Educaticn 



Alabana, 1975, 1979 
1st and 2nQ Quadren- 
nial Evaluation Coa- 
aissLon Reports 
Alabasa, 1983 
Challenge - Obligation - 
Opportunity: The laoera- 
tivo for Excellence in 
Higher Education 
Council of 21 



Alabasa 1983, 1987 
3rd and Uh Quadrennial 
Education Coacission 
Reports 



South Carolina, 1980 


South Carolina, 1985 


South Carolina, 1??8 


South Carolina Master 


Higher Education in 


Legislative Audit 


Plan for Higher 


South Carolina: An 


Education 


Agenda for the Future 






(AVAl 





The Master Plan 
for Florida Post- 



Report and RecoDoenda- 

_ tions of the Joint 

secondary Education Legislative and Execu- 
tive Coaaittee on Post- 
secondary Education 
Florida, 1985 
A Path to Excellence in 
Public Higher Education 
in Florida 



Florida, 1979 
A Call to Action 
(ABD) 



Vest Virginia, 1985 
Agenda for Action 
1935-90: Haster Plan 
of Goals and Service 
for Public Higher 
Education in Vest 
Virginia 



Vest Virginia, 1979 
Support, Perforaance 
and Protection of 
Higher Education in 
Vest Virginia 
(AED) 



Vest Virginia, 198^ 
Perforaance Audit of 
the Vest Virginia 
Board of Regents in 
conjunction with 
Sunset review 



Idaho, 1936 
Strategic Plan 
Directions for 
Higher Educat'^n 
in Idaho: The 
Next Five Years 



Idaho, 1983 
Higher Education in 
Idaho; A Plan for the 
Future (lACIl 



Idaho, 1983 

The Future of Higher 

Education in Idaho 

(ECS) 

Idaho, 198^ 
Northwest Regional 
Accreditation Report 



Ida::, 135: 
Ai 1' c ^^nri**"-* 

tc eiasir.e sta'.e 
joverRance and 
z-Kit] collele- 
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Colorado, 1932 
Sunset Review, 
Cosaittee on the 
Colorado Coaaission 
of Higher Education 



Colorado, i9?5 
Report of the 
Higher Education 
Coaaittee 



by legislative committees, four by state legislative performance 
audit or sunset reviews, four by a statewide board, and one by a 
regional accrediting association. Clearly a great deal of 
evaluation has taken place - - without mentioning the studies 
undertaken in the other 43 states! (In Appendix A a brief 
annotation is ^iven of three recent surveys by the Education 
Commission of the States regarding blue ribbon commissions and 
state assessment activities in higher education,) 

For each state, a background paper on the state system was 
prepared by a knowledgeable local person. These reports were 
extremely useful. They gave background on issues, actors, events 
and contexts which enabled the authors to gain an understanding 
of the particular characteristics of that staters situation. The 
understanding xvas then tested by interviews with 15-20 key actors 
who had been involved in the evaluations. 

The state reporters were : 

Alabama - Dr . Elizabeth French , Assistant Director for 
Academic Affairs of the Alabama Commission on Higher Education 
for nine years and coordinator for three of the four commission 
evaluations mandated by the legislature, 

Colorado - Dr, Charles Manning, Deputy Director of the 
Colorado Commission on Higher Education; he served uith the prior 
commission • for about a decade. 

Florida - Vice Chancellor Roy McTarnaghan, who had been 
Executive Director of the Virginia Commission on Higher Education 
before coming to Florida in his present position about ten years 
previously, 

Idaho - Dr, Milton Small, who was Executive Director of the 
State Board of Education from 1972 to 1983, 

South Carolina - Mr, Alan Krech, who has been a staff member 
of the South Carolina Commission since 1970. 

Washington - Dr, William Zumeta, a Professor of Public 
Administration at the University of Washington, who has studied 
and written about state government and higher education. 

West Virginia - Dr. Harry Hefflin, former President of 
Glenville State College, former Vice President and Acting 
President of West Virginia University. 

In five of the seven states the reporter was an active 
leader/participant in the state agency; in one state, the 
reporter was retired from institutional leadership. Only 
Professor Zumeta could be considered a fully neutral observer. 
All the observers were knowledgeable about the studies being 
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researched and they made the effort to be as objectixj as 
possible. Their accounts were checked against the reports of 
other participant observers. 

The interviewees were representative of the following 
categories : 

- executive branch 

- legislators and staff 

- state board of higher education lay members and staff 

- leaders from public and private colleges and universities 

- selected lay citizens (e.g., members of blue ribbon 
commissions) and journalists 

Alabama and Florida were visited by both authors to set 
common research procedures; the remaining five were surveyed by 
the authors separately, but accompanied by a colleague to assist. 
In Colorado , Aims McGuinness assisted in interviews and 
discussing background. Assistance with interviews was provided 
by: William Zumeta in Washington, Gordon Van de Water in Idaho, 
Steve Smartt in South Carolina and Susan Studds in West Virginia. 

In each state, multiple studies were reviewed, and while the 
general outline of the issues to be covered in interviews was the 
same for all states, questions x>;ere tailored to the particular 
state and person at hand. The interviews x^ere conducted off-the- 
record to preserve and inspire maximum candor. 

Information collected in the interviexv's included a 
description of the evaluation process, why it was initiated, who 
performed it, what questions the evaluation sought to address, 
i>;hat answers were proposed, and what recommendations xcere 
developed. The political climate was explored, and interviewees 
also addressed the short- and long-term impacts they observed. 
They were asked to give opinions on hox^ the evaluation process 
should be conducted, who should be involved, what the proper 
timing x^ould be and what questions would be most important. 

A small Technical Advisory Committee, shown in Appendix B, 
reviewed this study design early in the process and then looked 
at tentative findings. In addition, issues were discussed x^ith 
small groups from governors' staffs, legislators, state higher 
education executive officers, and institutional presidents. 

Con-^^'u^^ions and observations collected by these methods are 
report :^ the follotvj.ng chapters as input from ''respondents" or 
"intei 3s." Each chapter contains examples of the subject 



evaluation processes and the respondents' interpretations which 
helped establish patterns and form our recommendations. 



A Preview 

The research for this report led to four major conclusions. 
Chapters III through VI will elaborate on each conclusion and the 
evidence which formed it. They are previewed here in ascending 
order of importance and in descending order of our certainty 
about them. 

1 . State boards of higher education should engage in 
periodic self-evaluation as a means of self-improvement and of 
ensuring accountability to the state. 

2. Whether or not such board evaluations take place, some 
states have formally mandated performance audits or sunset 
re^^'.ews which include the state agency or board for higher 
education. Such audits/reviews normally focus too narrowly on 
legislative intent and pay insufficient attention to the broader 
questions of the boards* effectiveness in leading emerging state 
issues, particularly those related to quality and assessment. 

3. While some states have attempted to address these broader 
issues through the use of special or blue ribbon commissions, 
out-of-state consultants and/or ad hoc or standing legislative 
committees, only limited success can be proven for these 
processes. It is recommended that such processes be undertaken 
periodically rather than used only at a time of crisis when the 
agenda of radical restructuring and/or change of key personnel 
can create a threatening context for the proposed reforms. 

4. Finally, even periodic state system evaluation efforts by 
special commissions and/or professional consultants cannot 
substitute for ongoing systematic assessment of the effectiveness 
of higher education programs and institutions. 

State governors and legislators are asking state boards of 
higher education to cooperate with the state ' s institutions to 
produce evidence of effectiveness. This involves multiple 
measures and careful design of assessment efforts in order to 
ensure fairness and comprehensiveness. By leading this effort, 
state boards may increasingly become characterized as "evaluating 
boards " for the assessments they conduct themselves , or for the 
institutional evaluations they monitor . Where ongoing 
evaluations are developed, statewide boards must incorporate the 
results of such evaluations into their own planning, budgeting 
and program review activities . 
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Chapter II 



STATE EVALUATION PRACTICES 



Before the actual research f indings are presented , a 
framework for understanding the evaluation process in higher 
education should be constructed, along with a preview of the 
evaluation models analyzed in detail in later chapters. 

Evaluation Frameworks 

State level decision making in higher education occurs 
through two different processes. Both can operate at any one 
time, and both can affect any given decision. One is incremental 
decision making, which has a short time horizon and involves 
negotiation and bargaining between different interest groups. 
Most political decision making is of this type. The essence of 
an incremental decision is to use the available information to 
reach a decision that will satisf ice ^ the various interests that 
are involved. 

Comprehensive is the second variety of decision making. It 
^as a longer time perspective, is based on planning and extensive 
data collection and expertise { Schmidt lein , 1982). Comprehensive 
decisions are made in the context of management goals and 
priorities. Feedback and evaluation are integral to this 
process. Comprehensive decision making is widely used in the 
management of large and medium-sized public or private 
organizations. 

Evaluation formats differ for incremental and comprehensive 
decision processes . 

Evaluation usually occurs in incremental decision processes 
because a problem or crisis arises. The conventional wisdom 
among incremental decfsion makers is that "If it ain't broke, 
don't fix it." Assessment is not a regular part of most 
incremental decisions , which are judged largely on the basis of 
their ability to "satisf ice" the various consti tuent groups 
involved. Formal evaluation , when it does occur as the result of 
a crisis , emphasizes judgment more than information or analysis . 
The emphasis is on having key interest group viewpoints heard and 



*"Satisfice" as used here means to reach an agreement which lets 
the organization continue , even though no one got all they wanted 
from the decis ion. Budgets are almost always "satisf iced j " 
agreements are reached and programs proceed even though no one 
gets all they want. 



represented in the final decision. Evaluation may involve the 
advice of experts, a task force, special committee, or commission 
which will hold hearings, review evidence, and draw conclusions. 
The judgments of experts or influentials are usually more 
important than information or analysis. The goal is to get 
agreement; a secondary goal may be to solve the problem. An 
incremental evaluation must deal with people's perceptions of the 
problem, whether or not they deal with the problem itself. 

Comprehensive decision processes usually have evaluation 
built into the process itself. Ideally, comprehensive decision 
processes proceed from planning to decision, to resource 
allocation , program implementation, evaluation and feedback 
followed by modification. This is often called the "rational 
planning model," Assessment is either continuous (monitoring) or 
it occurs at scheduled intervals. The emphasis is on objective 
and comprehensive information and analysis. Expert judgment is 
important, but a good data base and analytic methods are 
essential. The goals of comprehensive evaluation processes are; 
improvement of ongoing operations, and avoidance of operational 
failures , 

States now use both incremental and comprehensive 
evaluations, but the introduction of comprehensive systematic 
assessment at the state level , is quite recent. Incremental 
evaluation is still the dominant procedure. 

Historically, incremental evaluations of state higher 
education systems were done by special study groups. Evaluations 
varied in the use of experts as staff or consultants. These 
evaluations relied on unpredictable levels of information 
collection; most of them used information which was readily 
available since there was .o time for elaborate studies or multi- 
year analysis. 

The strengths of the special study commission are that it 
permits (but does not ensure) objective appraisal, and it 
provides a broad perspective on complex issues. When influential 
people serve as evaluators, the recommendations usually get 
attention and carry credibility , 

The weaknesses of this type of evaluation are that it may 
have limited information to guide decisions, and there is no 
continuity or institutional memory about the issues to be studied 
because the evaluation process is not institutionalized. It is 
quite common to see successive commissions at five to ten year 
intervals examine the same problems and come up with similar 
recommendations to their predecessors , There is little 
opportunity for follow-up and others are left to adopt and 
implement recommendations , 
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Despite these limitations, special commissions are a primary 
type of incremental evaluation, as will be shown later. They fit 
well into the political culture and the way that most state-level 
decisions are made. 

They also fit well into the autonomous and decentralized 
nature of higher education. Public colleges and universities 
have a great deal of substantive autonomy in America. Most of 
the important educational decisions about uho is admitted, what 
is taught, and what standards are maintained are made at or 
within the institution by faculty or departments. 

Institutions operate primarily by incremental decision 
processes, although growth has made them increasingly 
bureaucratic, and more comprehensive decision making is beginning 
to take hold. 

When state agencies were established, they were expected to 
develop comprehensive plans, make decisions based on data and 
analysis, and use objective criteria for their recommendations 
about budgets and programs. As one agency head put it: "Good 
information and a statewide perspective are our special 
contributions to the policy process." Although most agencies 
adopted comprehensive approaches to decision making, they also 
had to operate in a political environment with a more incremental 
style. State agencies have had to use mixed strategies. For 
example, they x^ould develop a five-year fiscal plan, make budget 
recommendations on objective criteria, and also participate in 
the annual budget bargaining process in the legislature to fit 
the requests to the available revenues. 

The strengths of the comprehensive approach are that the 
state agency can make an important contribution to the state 
policy-making process — it can offer a long-range statewide 
perspective and analysis of budget and program issues based on 
systematic and comprehensive information. This contribution is 
valued in states where legislative and gubernatorial s x. fs have 
become advanced in professional skills, a trend x^hich is 
increasing . 

The weakness of the comprehensive approach is that it can be 
ignored or rejected unless it can fit its analyses Xv'±^hin the 
ongoing incremental -political decision environment . The agency 
must operate with a mixed strategy, and it cannot depend only on 
logic and rational analysis to make its ca&e. In addition, the 
cost of data and analytical work is much higher chan in the 
incremental process • 

As the state higher education agenda has shifted from 
concerns about growth and access to concerns about quality and 
efficiency, the states have been pushed to get more involved in 
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comprehensive ongoing evaluation, and specifically, assessment of 
institutional quality . 

Involvement of the state agency in assessment provides the ^ 
continuity necessary to assess statewide objectives with multiple 
approaches that can be repeated periodically. In addition, it 
can provide comparable information for all institutions ; 
comparabili ty will not result naturally f rom ins ti tut ional- level 
evaluations • ^ 

However, the comprehensive state approach to evaluation maj' 
lead to institutional compliance without motivation to internal 
improvement. The institutions may believe self-evaluation is ths 
proper path to improvement and resist any external evaluation, 
especially if it may present them in a negative light. The ^ 
unknown consequences of an external evaluation are always 
threatening to an institution's most valuable asset - its image. 

Formative and Summative Evaluation 

The two major purposes of evaluation are improvement and ^ 
accountability. The former, often called formative evaluation, 
is focused on how organizations improve effectiveness. Self- 
evaluation is the formative process favored by higher education. 
It keeps control of the evaluation process within the academy. 

Accountability is a summative and judgmental process and 
almost by definition, this evaluation must be done by an external 
party. Fiscal accountability is a well-established process in 
our society, but program and managerial accountability are newer 
evaluation objectives with less well-defined rules and 
procedures . 

i 

Higher education is involved in both self-evaluation and 
organizational accountability. The two kinds of evaluation are 
so different that it is difficult to serve both improvement and 
accountability objectives x^;ith a single evaluation process. 

Process or Outcome Evaluation 

Process evaluation tells us nothing about x^/hether the 
business or agency was successful or unsuccessful in achieving 
its purposes . As an example in fiscal accountabi li ty , process 
questions would be: were the proper accounts kept, and were ^ 
proper procedures followed . Outcome evaluation, by contrast , 
tries to determine success or failure, or the relative 
effectiveness of the organization as compared with another. 

The outcomes of higher education are difficult to measure. 
Due to the diversity of pos tsecondary education, the outcomes 
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sought by different institutions vary greatly. As a result, 
evaluation of higher education has focused on the processes being 
used and the resource inputs available. It has also folloxced the 
self-evaluation model in its accreditation process which 
evaluates institutions in relation to their stated purposes. 
Variations in institutional purposes and clientele mean that 
common and general standards for evaluation of all institutions 
or all students are virtually non-existent. This is regarded as 
a virtue in our decentralized individualistic society, but 
because evaluation is so relativistic , it is hard to develop any 
real understanding of how well the systems perform. Are they 
improving, remaining stable, or declining? 

Crisis ♦ Periodic, or Continuous Evaluation 

The timing of evaluations is important . Comprehensive 
evaluation processes give feedback at regular intervals to 
improve decisions on budgets, curriculum, admissions standards or 
other institutional functions and practices. Self -evaluation as 
a part of accreditation has been regularized on five- to ten- 
year cycles. This periodicity is appropriate for functions that 
need to change slowly and need only occasional "fine tuning." If 
there is no perceived need for basic change and if there is 
satisfaction with the system, periodic evaluations can even 
become formalities v^ith little impact. 

The pace of change in our society is quite rapid and 
external influences, particularly fiscal crises, sometimes 
trigger special evaluations. Studies done in response to fiscal 
crises focus on efficiency issues - how can the system be cut 
back without reducing services? They also tend to develop short- 
term solutions. Crisis evaluations may have a different agenda 
to evaluate than periodic or continuous evaluation processes; 
they often focus on change of structure and/or key personalities. 

State Evaluation and Institutional Evaluation 

Institutional evaluation is focused on the performance of 
individuals - students and faculty . State evaluation , by 
contrast, is concerned with the assessment of broad program 
area?, institutions, and the effectiveness of the entire system. 
The state is concerned with the availability of opportunities for 
citizens, the extent to which the system serves the diverse needs 
of different client groups, and the ability of graduates to 
contribute to the economy and future development of the state. 

Even x.;hen the evaluation needs of the state and the 
institution are similar, the institutional data will not be 
useful at the state level withou t special summarization . States 
want to compare institutional results from year to year, between 
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institutions, and between themselves and other states. The need 
for outcome information requires comparability of measures across 
institutions, between states and over time. These needs will j 
require more uniform measures and summarization of data for 
multi-year periods. 

The Major Models of Evaluation 

Until the last two decades, most states left to the 
institutions the responsibility for assessment of programs, 
personnel, and performance of higher education. As an exception, 
the lieu York State Regents have long had formal responsibility 
for accreditation of the public and private institutions in their 
jurisdicti on, but only since World War II have other states 4 
developed formal program reviex>7 processes for public 
institutions. Many states have been minimal participants, merely 
requiring that institutions have a published process and leaving 
the responsibility for the review at the institutional level 
(Barak, 1983). Broader evaluations of institutional 

effectiveness (comprehensive in nature) have been even rarer, ^ 

Nevertheless, there are a variety of procedures which have 
been used to assess state agencies and state systems. The major 
models addressed in subsequent chapters are: 

1 . Self-evaluation ^ 

2. Performance Audit/Sunset Reviews 

3. Special study groups or commissions 

4. Comprehensive Reviews and Assessments 

Self -Evaluation ^ 

The ,7\ost fundamental and traditional form of evaluation is 
that of self-evaluation. As Chapter III explains, self- 
evaluation can take many forms - from informal self-evaluation 
conducted at a board retreat by a state board of higher education 
to formal exercises with out-of-state participation. These self- ^ 
evaluations tend to be formative in nature rather than summative, 
but serve useful purposes within those policy limits. 

A variation on board self-evaluation could be to broaden the 
regional accrediting processes to include evaluation of statewide 
boards. This was done by the Northwest Association of Schools | 
and Colleges for the Idaho Board of Regents and will be discussed 
below. 

The accreditation process has been an evaluation vehicle for 
system performance in other states. Maryland's Board of Higher 
Education made a formal agreement with the Middle States ^ 
Accrediting Association for participation in the periodic 
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accreditation of Maryland public institutions. New York's 
Regents accredit both public and private institutions. 

Self-evaluation against an institution's own goals can be 
use-^ul to a motivated institution; however, accreditation is not 
particularly'- helpful in responding to the state's accountability 
interests. Accreditation reports are not public, which sharply 
limits their ability to serve the accountability process. State 
officials are skeptical of the institutional self -evaluation 
process because they often see self-regulation fail in their own 
professions. For these reasons, many states are developing 
institutional evaluation procedures that are independent from the 
traditional accreditation process. 



Performance Audits 

State legislatures are concerned about their responsibility 
ior ensuring program accountability. In the 1960's and 1970's, 
many states developed program audit staffs to examine the 
effectiveness of state agencies. Evaluations of higher education 
agencies by audit staff have occurred in many of those states. 
Some program audits of higher education are done as part of 
periodic "sunset** reviews which determine whether the agency 
should be continued. Other audits have besn ad hoc examinations 
initiated by legislators. 

Audits are usually narrow in scope , and have a tendency to 
become management or financial audits which examine what the 
agency is doing to follow legislative intent. They usually do 
not examine the operation of the entire higher education system 
and usually do not assess the state board's role in leading the 
system, setting policy and addressing broad issues. 



Special Study Commissions 

Prior to development of most statewide higher education 
agencies, special study commissions were the dominant model for 
examining statewide higher education systems • These were '*blue 
ribbon commissions" of lay citizens or legislators, or a 
combinati on of both. The studies served many purposes! future 
planning , resolving disagreements , setting public agendas for 
action, fact-finding and analysis, and evaluation. 

As state agencies assumed responsibility for planning , 
program approval and resource allocation, ad hoc statewide 
studies by commissions decl ined in frequency . Continuous 
planning by an agency which had the capacity to follow up on its 
decisions was believed to be preferable . However , the master 
plans developed by state agencies involved limited evaluation of 
existing institutions and programs. Most were future-oriented 
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plans designed to chart a course for higher education 
development. Assessment was not usually a major purpose '>f these 
planning efforts. In the 1960's, higher education was growing 
rapidly and state agencies were trying to control and direct that 
growth with their plans; assessment of quality received little 
emphasis . 

As the economic , educational and political environment 
became more turbulent in the late 1970^Sf concern about the 
adequacy and effectiveness of state education systems grew among 
governors and legislators. Some questioned the ability of 
statewide agencies to deal with these complex issues and there 
was a resurgence of ad hoc blue ribbon commissions to examine 
higher educations systems (Johnson and Marcus, 1986). These 
special commissions are used for several reasons: 

1 . If the state needs to examine all levels of education 
there is usually not a single agency which could appropriately 
conduct the study. Most states have separate boards for K-12 and 
post secondary institutions . 

2. If the state higher education agency does not have the 
confidence of the legislature and governor because the board has 
been seen to be ineffective in dealing with policy issues, the 
state will look elsewhere for advice. 

3. The state agency may be seen as an advocate for higher 
education and a more objective assessment is needed. This will 
be particularly true if state leaders think a change in 
governance structure is necessary. 

4 . When the state faces a major crisis in funding or 
quality, it is important to get proposed changes on the public 
agenda where they can be supported by state leaders. A special 
commission of influentials may be able to accomplish this most 
readily . 

About half the states have appointed special commissions to 
study and make recommendations about higher education ( or all 
levels of education) during the past five years (ECS , 1986 ) . The 
commiss ions are more f requent in states which have experienced 
economic downturns , but that is not the only reason for their 
appeal to state governments. 



Comprehensive Evaluation 

Recent interest in higher education reform and improvement 
has led public officials to urge assessment of system outcomes 
and effectiveness in meeting state objectives. Assessment is 
seen as an integral and necessary part of program improvement 
efforts. States like New Jersey, Tennessee, Missouri, Florida 
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and Arkansas are responding to this new mandate. The push for 
assessment has not been institutionalized in most states and 
state agencies are still trying tc determine how assessments 
ought to be conducted. The governors have jointlj'- recommended it 
be ongoing, that it involve institutional leadership, and that 
multiple measures of outco'nes be a ^.-art of the assessment (Time 
for Results 1986) . 

An assessment system that would satisfy those objectives 
would examine student learning outcomes and the readiness of 
graduates to fill professional and technical jobs. It would 
report regularly on the extent to which each institution is 
meeting state objectives. The evaluation system should be 
structured to make the information useful in the planning and 
budgeting process. Because evaluation data can be unfavorable, a 
comprehensive evaluation system of this type will be threatening 
to some programs in many institutions , and to moiit programs in a 
few institutions. It will also require institutional leaders to 
pay attention to objectives that are being measured by the 
evaluation system. Designing an assessment system that meets 
state accountability needs ai^d also provides positive incentives 
to institution? to improve themselves ^s the real challenge. 

A recent survey of state level assessment activity by SHEEO 
and ECS shows that agencies in 30-35 states are involved in some 
form of ongoing assessment program. In a majority of these 
states, the state agency is monitoring institutional assessments, 
but each institution develops its own evaluation program; some 
states set state guidelines for assessment. In several states 
there is an implicit goal of having institutions do credible 
evaluations to keep the legislature or another state organization 
from taking over the process. Most states are trj'-ing to assess 
institutional and program effectiveness in conventional xvaj's such 
as peer review. There is, however, a new emr)hasis on measuring 
student outcomes. About a dozen states are at various stiJges in 
defining and developing methodologies for outcome assessment. 
Some states are taking several years to involve ins t i tut ions in 
developing measures and implementing reporting systemtr. 

Summary 

States are entering a new phase where many state agencies 
are encouraging, assisting and monitoring institutions as they 
establish assessment systems . At the same time , states are 
continuing to do program audits and set up special commissions to 
study and evaluate higher education. State leaders still turn to 
prominent citizens and outside experts for advice and 
consultation on education systems even if a comprehensive 
evaluation system is in place or being developed. This is 
because political leadership has always availed itself of advice 
from many sources and comprehensive evaluation x^ill not have all, 
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if even most, of the answers state leaders seek. The combiaed 
environment of incremental and comprehensive decision making 
perpetuates the use of varied evaluation models. 
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Chapter III 
SELF-EVALUATION BY THE STATE BOARD 

This study found that self-evaluation can be valuable in 
improving state higher education agency operations. Houever, it 
does not obviate the need for comprehensive evaluation of a 
staters system of higher education. Inasmuch as improvement of 
the effectiveness of a state board of higher education may be 
thought to lead to improvements in the overall state system, a 
linkage can made between the two processes. 

This chapter will focus on board self -evaluations which can 
run the gamut from highly informal processes to formally 
structured periodic events. The examples given here demonstrate 
the various levels of formality and the diverse formats which can 
be construed as self-evaluations of boards. 

A. Processes . Boards initiating self-evaluation can: 1) 
develop internal guidelines and processes; 2) use a set of 
self-study guidelines and criteria published by the Association 
of Governing Boards (AGB) (one set for state coordinating boards 
and another set for state consolidated governing boards); 3) hire 
a consulting firm to undertake the evaluation; or 4) create an 
ad hoc group of consultants who will develop a set of 
procedures. Selected cases will illustrate each alternative. 

1. Informal Evaluation . At the minimum, a state boax*c can 
set aside some time from its regular operating agenda to take 
stock of how it is performing various missions. The success of 
such exercises is dependent on advance planning, and is often 
conducted as a one- or txvo-day retreat. The Arizona Board of 
Regents generally reported that such activities are helpful and 
constructive • 

In some states , however , sunshine laws inhibit board 
self-evaluation efforts; boards may feel that candid 
self-criticism in public could be used by critics to weaken the 
board. In some cases board leadership may be hesitant to 
initiate a public board evaluation process because they don't 
think it is appropriate or productive to discuss personnel and 
other sensitive matters in public. 

2. Association of Governing Boards - Guidelines . The AGB 
reports that, by far, institutional and multi-campus system 
boards are the heaviest users of AGB self -evaluation kits 
containing evaluation cri teria and guidelines on how to undertake 
the process . The AGB also nominates qualified persons to serve 
as "Board Mentors" during the recommended sequence which includes 
a self-study and careful follow-up. 
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The AGB indicates that three state consolidated governing 
boards have used the self-evaluation k^ts since 1983 (Iowa, 
Mississippi and Nevada) • We heard a generally favorable set of 
reactions from one of the states, with cautions about timings- 
proceed during stable conditions as opposed to during a cri^sis 
state--and the importance of selecting sensitive Mentors. 

3. Consulting Firm . In 1986, the Minnesota Higher 
Education Coordinating Board decided as part of its adopted 
*'Board Objectives for 1986" to: 

conduct its own mission study with a 
performance audit of the Board, staff, 
projects and responsibilities , including a 
specific examination of whether the resources 
devoted to Policy Analysis and Development 
are sufficient to meet the ]Boar.d's 
responsibilities in these areas, and to 
develop , as needed, specif ic proposals 
including budgetary and legislative changes 
to enable the Board to exercise discretion in 
setting its ann'\al management plan and to 
enable the Board to continue to develop 
innovative and new proposals and projects 
{ 1986) . 

The audit was conducted by a four-person team assembled by 
the M & H Group Inc. of Boulder, Colorado, with further help from 
a three-person Advisory Panel. The audit team spoke only to 
board members and staff and did not tap the opinions of persons 
in higher education institutions or in state government. The 
process occurred between April and September 1986 . 

The 63-page report (plus appendices) ends with findings and 
recommendations concerning: 1) board-related issues; 2) 
organization issues; and 3) management issues. 

The board executive director reported that the board felt 
comfortable with the results because the board itself issued the 
Request for Proposal that initiated the process. 

4. Ad Hoc Consulting Group . In 1982 the Maryland State 
Board for Higher Education (SBHE) asked four persons from out of 
state to conduct an evaluation of the SBHE's general 
effectiveness. A Columbia University Teachers College 
professor, the President of the University of Kentucky, a Kew 
Jersey legislator, and a former SHEEO (author Folger), 
constituted the evaluation group. In this case, the team was 
given a free hand to develop its own agenda. It interviewed 
state board members, board staff, and key actors in higher 
education and state government (including the governor). 
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The 46-page report xcas divided into analyses of nine board 
functions each wiLh sections* on "views presented to the team," 
"findings," and '^reconmendat ions • " The report x^-as used 
essentially as an internal document, albeit one shared with the 
governor. Reportedly, the board and staff found the 
recommendations useful t 

5. Legislatively Mandated Periodic Self-Evaluation . This 
variation will strike some readers as not qualifying for a 
self -evaluation label , but it is at least in part a 
self-evaluation. For example, enabling legislation creating the 
Alabama Commission on Higher Education in 1969 included a 
provision requiring an evaluation of the effectiveness of Hie 
Commission every four years by a team including at least three 
persons not associated wi th higher education in Alabama . 

Why could a state-mandated evaluation be termed 
self-evaluation? Primarily because the Commission plays the 
major role in the selection of the evaluators, the reviews are 
periodic (non-crisis), and are formative in nature. 

The distinction between formative and summative evaluations 
is based on who initiates the assessment and what its major 
purpose is. Most of the various forms of state evaluations 
described in later chapters are either explicitly summative {e./|. 
a sunset reviex^?) or are ad hoc evaluations emerging during some 
time of board or state crisis x^hen the agency's future is at 
least implicitly at risk and therefore construed as summativ*>. 

Alabama has nox^ been through four c^'cles of its 
state-mandated evaluation (1975; 1979; 1983; and 1987) and each 
report has been, clearly formative* On the first cycle, three 
<^valuators were from out-of-state; four from Alabama. For the 
second cycle, four out-of-state reviewers (including Berdahl ) 
worked with five Alabamians. During the third evaluation cycle- a 
blue ribbon planning group, the "Council of 21", x^-as formed of 
prestigious state citizens. The formal evaluation xcas undertaken 
by four out-of -staters ; the commission did not feel it necessary 
to involve Alabamians beyond those in the blue ribbon group. 
Finding this set up successful, the CommissioK stayed with the 
pattern of four out-of-state evaluators for the fourth cvcle in 
1987. 

It is important to note that in each case the Commission 
chose the persons who t^ere to evaluate it, making the process 
into a de facto formative evaluation. '.?he Commission did solicit 
nominations from the institutions, but the persons chosen x^ere 
those in whose judgments the Commission had confidence. 

In 1975, the first team raised a sensitive point. Those 
appointed were given a charge to "call them as they saw them", 
but presumably there was an anticipation that the judgmen ts xcould 
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be balanced. People appointed by and, in the case of 
out-of-state professionals, paid by Commission leadership could 
feel inhibited about being too critical in public. For this 
reason, plus the additional important purpose of increasing the 
executive and legislative branch attention to evaluation 
recommendations, the 1975 team formally urged that future teams 
be constituted by a revised process. They suggested that the 
governor, the presiding officer of the Senate, the presiding 
officer of the House, and the Commission each appoint an Alabama 
citizen not presently connected with postsecondary education in 
the state, and that these four then (with the advice of the 
Commission) choose the out-of-state team members. This 
recommendation was not accepted and the board^s original 
selection process has continued. 

Jerry Miller, a member of the first team, analyzed the 
composition and internal relations of his group. The combination 
of in-state and out-of-state was good because the "locals" could 
make the team more sensitive to Alabama issues, and later 
convince citizens and state officials of the need to implement 
the recommendations. The out-of-state members could bring 
national perspectives to Alabama issues as well as knowledge 
about the performance of comparable commissions in other states. 

Miller added that it was important for all seven members of 
the group to be on equal f ooting--there were no "primary members** 
or "consultants specializing in this or that topic." Thus 
discussions were wide-ranging and free-wheeling and a group 
consensus emerged (Miller, 1975). 

The first evaluation team encountered only modest reactions 
to its 30 recommendations. Although a later report judged that 
21 of the 30 recommendations had been favorably responded to and 
another two partially, the important recommendations to turn the 
Commission's advisory powers into regulatory powers were not 
heedeji. This may have been due, in part, to political relations 
between Commission members and the lieutenaxit governor vho later 
used his considerable powers to persuade the legislature not to 
strengthen the Commission but to cut its budget significantly 
instead. A larger variable which affected not only the 1975 
recommendations but all subsequent ones as well was that the 
Commission was in a otate with a basic political culture not 
receptive to objective rational planning . Generally , the 
political forces (particularly those operating during the long 
periods when George Wallace served as governor) were not willing 
to base budget decisions, including many supplemental individual 
items, on so-called "rational planning". And the leaders of the 
more powerful higher education institutions felt that since they 
had to play the political game as individuals anyway, a stronger 
commission was not in their self-interest. 
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The second evaluation stud}'" Kas marginally more successful 
in convincing the legislature to pass a 1979 bill strengthening 
the Commission. The second team disseminated its findings by 
printing several thousand brochures summarizing its full report 
and distributing these to organizations like the League of Women 
Voters and the Chamber of Commerce. 

The legislature, however, included an extra message in the 
final bill: it required the upper house to confirm the 
Commission's executive director every four years I 

It was perhaps no coincidence that the third evaluation in 
1983 emphasized the need for the Commission to improve its 
relations with the executive and legislative branches of state 
government . 

The 1987 evaluation process is too recent to evaluate 
impact. Its key recommendation was for the Governor and 
legislative leadership to create a state blue ribbon task force 
to recommend governance and finance policies for Alabama higher 
education — to the end of the centurj''! The chair of the team 
spent time in the state after the report was published in order 
to present it to legislative and executive branch personnel. 

In sum, the majority of the persons interviewed in Alabama, 
including both the present and former executive directors of the 
Commission , supported the periodic state -mandated evaluation 
process. However, many respondents commented on the limitations 
imposed by this particular state's political environment. 

6 . Self-Evaluation Through Accredi tat ion . The presence of 
a common member on both the Idaho Board of Regents and the 
regional accrediting association, the Northwest Association of 
Schools and Colleges, gave rise to a new form of board self- 
evaluation. The dual membership probably inspired the suggestion 
that the regional accrediting body assess the role of the 
statewide board while re-accrediting the four Idaho public senior 
institutions . 

The Northwest Association accepted the invitation and 
created four campus teams and a fifth team for the system. The 
latter was composed of persons from adjoining states with senior 
administrative experience in statewide systems. The chair was 
the Chancellor of the Oregon State System of Higher Education and 
a former president of one of the Idaho senior institutions. 

Just as individual campuses are required to , the Idaho Board 
of Regents developed a self-study. The system accrediting team 
spent time in the state capital discussing issues with the IslY 
members and staff of the state board, visited each institution, 
and talked with the campus accrediting teams. 
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The resulting report (1984) was formative evaluation at its 
best. The recommendations called for a series of measures to 
strengthen the Idaho Board of Regents, but always with a strong 
flavor of understanding the constraints within which the board 
had to operate. 

Jim Bemus, Executive Director of the Northwest Association, 
accompanied the system team, ex officio , and concurred Xvith the 
way the system team operated. He pointed out that it was easier 
to undertake a comprehensive system evaluation like this one in a 
state like Idaho with only four senior institutions than it would 
be in states with many more public colleges. He also noted that 
the Northwest Association had to subsidize the system team 
expenses as the more demanding time and travel needs could not be 
met by normal fees. 

As might be expected from an accreditation study, not many 
persons outside academe were aware of the accredi tation report 
on the system. Janet Hay, the dual member of the Regents and the 
accrediting association who was th^ initiator of the process, 
later became a state legislator and used the report (and another 
blue ribbon commission report mentioned in later chapters) to 
constitute the agenda for a 1985 legislative commission inquiry 
into higher education governance. But partly because regional 
accreditation works quietly and confidentially, and partly 
because many legislators are fairly skeptical about the 
accrediting movement's ability to be rigorously critical of 
higher education, this Northwest process must be labelled a 
formative evaluation with the strengths and the distinct 
limitations that routinely accompany the accreditation process. 

B. Conclusions . From these examples, what can be 
generalized about the board self -evaluation process? 

First, board self-evaluations are considered helpful in 
improving board performance and, thereby, possibly also system 
effectiveness. Evaluation processes which use mentors or 
external groups seem most likely to be helpful. 

Second, if there are serious criticisms about the 
effectiveness of the state board or the state system, or if there 
are serious political divisions within the state board, then 
self-evaluations will not be sufficient to restore credibility. 
There is understandable skepticism about the rigor of 
self-evaluations, and any process that is designed to influence 
state leadership must include summative dimensions not usually 
present in self -evaluation . 

Third, the several more elaborate evaluations of the Alabama 
Commission indicate that while the board could improve its 
internal operations as a result of these formative assessments, 
it has not been able to use them to persuade state political 
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leaders to rely more on the comprehensive planning and evaluation 
process • 

Fourth , the evaluation of state boards by regional 
accrediting associations has potential for internal formative 
evaluation. However, it would be difficult to attempt it in 
states with many public institutions. The ultimate value of such 
an exercise may lie in the greater appreciation of the values and 
problems of statewide governance and coordination gained by the 
accrediting agency in the course of conducting the evaluation. 
One of the criticisms leveled at regional accreditation of single 
campuses is that such evaluations usually pay insufficient 
attention to the campus^ role in its multi-campus and/or state 
setting . State boards that invite regional accrediting 
association evaluation may be getting two values for the price of 
one ! 
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Chapter IV 

ACCOUNTABILITY, PERFORMANCE AUDITS AND SUNSET REVIEWS 



A. The Changing Pattern of Accountability 

The development of performance audits and sunset revieis^s 
over the last 20 years represents a major development of the 
accountability movement in state government. When state 
government x^as simpler and smaller , legislators set budget 
priorities and judged agency effectiveness without much 
systematic information. As state government expanded in size and 
complexity, legislators sought new ways to assess the performance 
of state agencies . Regular fiscal audits (which were established 
processes in all states by the time World War I began) were 
sometimes supplemented by management audits concerned with 
efficiency and compliance with legislative intent. Most state 
agencies operated in a context which assumed indefinite 
perpetuation . 

In the late 1960's and early 1970*s, legislators began to 
focus on the need for measurement of outcomes, and the use of 
cost/benefit analyses as a better basis for deciding among 
competing policies. A Legislative Program Evaluation (LPE) 
section emerged as part of the National Conference of State 
Legislatures and LPE units appeared in over 40 states during the 
two decades. 

In some cases the LPE unit was a new legislative creature, 
such as the Joint Legislative Audit and Review Commission in 
Virginia where a multi-disciplinary team undertook performance 
audits of various Virginia state agencies and reported to a 
bipartisan joint legislative committee. 

In other states, the program evaluation function was added 
to the state auditor's traditional jurisdiction and the 
post-audits were broadened beyond fiscal and management 
evaluations of legality and efficiency to in^^.ude issues of 
ef f ectiveness . An effectiveness evaluation necessarily involves 
questions of goals, purposes and values, and these cannot always 
be derived from statutes. Thus, for those agencies selected in 
any particular year for performance audit, LPE staff would try to 
develop performance indicators to determine the extent to which 
the presumed goals were being met. Evaluators look for 
indicators that are quantitative in nature, but also use 
interviews, questionnaires, and other attitude measures. Draft 
audit reports are normally shown to the agency in question which 
has an opportunity to correct factual mistakes and to comment on 
disagreements in judgment. The final report may contain the 
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audit report, the agency *s reactions and the audit staff's final 
rejoinders . 

Beginning with Colorado legislation in 1975, most states 
have enacted some form of sunset legislation wherein state 
agencies are scheduled for review and possible termination by a 
certain date unless explicitly renewed. In some states, all 
state departments are subject to sunset review, and in other 
states , speci f ic agenoies are targeted . Sunset evaluation , as a 
tool of acoountabili ty, overlaps considerably with the 
performance audit process and occasionally the same staff is used 
for both functions. 

B. The Application of this Accountability to Higher Education 

While many may applaud new accountability patterns as a 
significant advance in the art of government , there are those in 
higher education who argue that the outcomes of higher education 
are less susceptible to this kind of measurement. Two case 
studies of state performance audits of higher education, 
illustrate typical controversies surrounding efforts to apply 
such evaluations: the community college system in Virginia and 
the University of Wisconsin system (Berdahl, 1977). Even though 
each of these assessments began as a comprehensive program audit} 
each ended up as primarily a more narrow management audit. 

This management di mens ion of the audit is likely to come 
most into play when performance audits/sunset reviews are turned 
toward the operations of state boards. Management audits of 
state boards tend to involve an analysis of the extent to which 
they are achieving their officially established purposes and hoi: 
well they are conducting their primary functions . In this 
chapter, four examples of performance audits, sunset review's or 
accountability evaluations will be presented. They reveal the 
probable outcome of auditing a state board independent of larger 
system-wide issues and structures. 

1. Connecticut and South Carolina . The first two 
performance audits of state boards occurred in Connecticut in 
1977 and South Carolina in 1978 when criticisms of the boards in 
those states prompted legislative interest. Although both ^-ere 
done by the LPE state agency, there were big differences in the 
two reports . This report concentrates on the South Carolina 
process which was more elaborate. 

The Connecticut study Strengthening Higher Education in 
Connecticut (April 1977) gave considerable evidence of having 
been conducted under time pressure. Some of the original charge 
to the Legislative Program Review and Investigations Committee 
x^as not included in the report, and an "Agency Response" normally 
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included in LPE reports was omitted. ^ The weakness of the report 
as an evaluation document was that it contained no explicit 
framework for evaluation of the statewide board in its existing 
form. Instead, general conclusions and recommendations were 
presented about the need for stronger central leadership, a 
stronger board role in planning, budgeting and program review and 
a better management information system. These judgments may or 
may not be correct; the point is that the document does not 
connect them to any criteria for evaluation. The proper balance 
between state oversight, coordination and accountability on the 
one hand, and institutional initiative, autonomy, and 
responsibility on the other is not easy to determine. Without a 
framework for assessment , the likelihood of divergent judgments 
is great. 

In contrast, the South Carolina process was much more 
careful both in devising a conceptual framework and in the 
lengths to which the staff went in order to acquire relevant 
evidence. Ironically, this elaborate process evaluated a board 
that was altered just before the audit report came out! A bill 
strengthening the Commission and removing its institutional 
representatives passed three months before the audit report was 
published with similar recommendations. Nevertheless, the 
process merits our attention. 

In 1977 the Legislative Audit Council (LAC) in South 
Carolina was asked by the Speaker of the House to evaluate the 
Commission on Higher Education (CHE). The subsequent report. 
Management and Operational Review of the South Carolina 
Commission on Higher Education (June 14, 1978), recommended 
changes in CHE membership and in several aspects of CHE 
operations in planning, budget review and program reviex.;. 
Although the report was controversial in that it explicitly 
disavowed any intention of describing "CHE accomplishments,** the 
seriousness of the LAC effort was clearly demonstrated not only 
by the creation of elaborate evaluation criteria, but also by the 
extensive additional activities which were undertaken . Covering 
CHE development since its establishment in 1967, the LAC audit 
staff interviewed 56 persons, mostly from the ranks of higher 
education, observed CHE meetings and committee functions over a 
nine month period, visited coordinating agencies in Alabama and 
Tennessee, inspected agency plans from 17 other states, employed 
several out-of-state consultants (including author Berdahl) to 
evaluate specific aspects of CHE operations, and conducted a 



1 The study recommended" the replacement of the existing 
coordinating board by a stronger board with changed membership. 
The state government subsequently changed the board functions and 
authority; but the link between the audit report and the 
legislative action is not clear* 
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user-survey of 234 persons in the State concerning the quality 
and usefulness of CHE studies and reports. The LAC was a 
relatively new agency in South Carolina (it was established in 
1974 ) and it was trying to demonstrate its professionalism by 
conducting a careful performance audit. 

The evaluation criteria considered goals and objectives, 
planning, budgeting, program evaluation, and overall adequacy and 
effectiveness of the Commission. For each of these headings the 
audit staff developed research questions, methods, measures and 
criteria. (A later evaluation of the South Carolina Commission 
occurred as part of a special study of the whole state system by 
the AVA group in 1986. This study will be analyzed in Chapter 
V.) 

2. Colorado . The Colorado Commission on Higher Education 
x^as established in 1965, strengthened in 1970, and had een its 
budget and staff severely cut in 1975. When a bill in 1977 
strengthened the Commission, the bill included a sunset provision 
for the year 1983. 

The first step of the Colorado sunset review is a self- 
study. An outside sunset review committee is appointed to hold 
hearings, review the self-study, and to publish its own report. 
The State Auditor's staff then conducts an independent 
performance audit. In this review, a seven person citizen 
committee chaired by a former legislator was formed to critique 
the CCHE report. 

The resulting report (Sunset Review Committee, 1982) 
analyzed criticisms of the Commission in 10 different categories. 
It found the Commission's activities justified, but urged greater 
sensitivity to the need to keep the institutions informed of the 
Commission ' s positions . 

The state auditor's Report (July 1982) came out three months 
later and endorsed continuation of the Commission but recognized 
some substantial problems with clarity of authority, inadequate 
information systems, and conflict with governing boards. 

For each of ten functions (not the same ten as the Sunset 
Review Committee) the audit staff answered the same four 
questions : 

1. Is there need for this function on a centralized basis? 

2 . Is the Commission the appropriate organization to 
provide the function? 

3. How has the Commission addressed the function? 

4. What improvements are needed? 
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To gather the necessary information, the audit staff used 
the following sources: 

State Law 

Legislative and other public hearings 

Commission reports, studies and other publications 

Site visits to various institutions 

Interviex^7S with legislators, students, governing board 
members and staff, and college personnel 

Questionnaires (reported in their Appendices) to governing 
board members, researchers and student financial aid officers. 

Clearly, in Colorado as in the 1978 South Carolina audits 
considerable care was taken to undertake a comprehensive audit. 

Most of the Colorado participants interviewed for this study 
gave the overall sunset review process low marks for 
effectiveness. Th-^s judgment was based on the fact that the 
process had not led to the termination or "sunset" of any a^enci 
except the Commission on the Status of Women, and that the 
process, as it evolved in Colorado, had not led to many 
constructive changes in agencies or any substantial saving of 
money. The agency self-study was seen as a way for the agenc> to 
shape the audit report in a direction that would be favorable to 
the agency, and to build up support for the value of the agency. 
The respondents felt that the self-study of the Commissiox: was a 
good example of presenting the agency in a favorable xcay (which 
may have been justified) and that the audit did not (and probably 
could not) deal with some major policy issues facing uhe state, 
including governance questions. The auditor's report did call 
for the legislature to undertake further studies of the governin<^ 
patterns in Colorado. The Commission endorsed that 
recommendation, and that was, in fact, the scenario Xv^hich 
transpired. The Commission's life was extended and the 
legislature authorized a special citizen committee to study and 
make recommendations about restructuring Colorado higher 
educat ion--issues the sunset review sidestepped. This study Xvill 
be discussed in Chapter V. 



3. West Virginia . The West Virginia Board of Regents Xvas 
established in 1969 and was first evaluated in 1979 by an 
external study by the Academy for Educational Development (AED) 
commissioned by the state legislature. This AED study is covered 
in a later chapter. Here it is sufficient to note that the 
legislature did not accept the primary AED policy recommendations 
to abolish the Board of Regents and to replace it uith a 
statewide coordinating board with segmental governing boards for 
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the university, state colleges and community colleges. The 
legislature did, however, act in 1981 to strengthen the Board of 
Regents in several ways recommended by the AED as a second policy 
option. In 1979, the legislature had enacted a West Virginia 
sunset law, which scheduled the Board of Regents for a review in 
1984, 

The Legislative Auditor's Office which conducts the sunsat 
evaluations has a .small staff and the schedule of state unics to 
be assessed each year is very demanding . Ten i eviews wore 
scheduled with the Regents in 1984; most years had 10-12 agency 
reviews scheduled • 

For each agency reviewed, the audit staff are to try to 
answer the following ten questions: 

(1) The nature of the objectives intended for the program 
or entity and the problem or need which it was intended to 
address, the extent to which the objectives have been achieved, 
and any activities of the entity or program in addition to those 
granted by statute and the authority for these activities, 

(2) The extent to which the government entity or program 
has operated in the public interest and the extent to which its 
operation has been impeded or enhanced by existing statutes and 
any other circumstances bearing upon the governmental entity's or 
program's capacity or authority to operate in the public 
interest, including budgetary, resource and personnel matters; 

(3) The extent to which the jurisdiction of the entity or 
program duplicates those of other entities and programs and the 
extent to which the entity or program or its activities could be 
considered with others ; 

(4) The efficiency with which the agency operates; 

(5) The extent to which the governmental entity or program 
has recommended statutory changes to the legislature which would 
benefit the public ; 

(6) The extent to which the entity of program issues and 
enforces rules relating to potential conflicts of interest of its 
employees ; 

(7) The extent to which affirmative action requirements of 
state and federal statues and constitutions have been complied 
with by the governmental entity or program; 

(8) The extent to which the governmental entity or program 
has encouraged participation by the public in making its 
decisions ; 



31 



(9) The impact in terms of federal intervention or loss of 
federal funds if the agency is abolished; 

(10) The extent to which the governmental entity or program 
has caused an unnecessary burden on any citizen or other 
governmental entity or program by its decisions and activities. 

The Legislative Auditor's Office published its Performance 
Audit of the Board of Regents on June 30, 1984, answering as bost 
it could the ten questions listed above. In this task it was 
aided by both the Board's answers to the questions and by the 
judicious use of lengthy quotes from the 1979 AED evaluation 
which now had a second life due ''o the overworked condition of 
the small staff. 

The Joint Committee on Government Operations, to whom the 
audit was addressed, agreed to recommend continuation of the 
Board but, instead of a fiv^ year extension, called for another 
sunset review before July 1, 1988. This highly qualified vote 
of confidence reflected less the relatively muted criticisms of 
the Board in the audit report and more the volatile political 
climate in the state. As an example ol i;his volatility, Governor 
Moore in 1986 recommended abolition of the Board and its 
replacement by a three-person, paid, full-time coordinating 
board, with governing boards again at the institutional level. 
The legislature (majority of Democrats) did not accept the 
governor's (Republican) recommendation. The Board is undergoing 
its 1987-88 sunset review, 

4. Washington , Performance audits of state agencies bes?an 
in Washington as early as 1972, and a sunset law was passed in 
1977, The legislative audit staff estimates that it spends about 
30% of its time on sunset reviews and the other 70% on management 
and policy audits. The various audits are similar with one 
obvious distinction - the sunset review has an explicit 
continuation/termination issue to be faced. The legislative 
audit process in Washington has been criticized for being too 
preoccupied with auditing small agencies performing obsolete 
functions, and not spending enough time evaluating the major 
functions of state government where more than 95% of state funds 
are expended. 

This criticism did not apply to the 1984 audit of the 
Council for Pos tsecondary Education (CPE), for it was a 
substantial and costly agency with major functions. 

The Council on Higher Education, as it was originally 
called, was established in 1969 as an advisory group with 
planning responsibility but without regulatory authority. It 
solidly established itself with strong legislative support, high 
quality gubernatorial appointments , and astute executiv-e 
leadership. It was reorganized as the Council on Pos tsecondarv 
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Education in 1974, and most of the college presidents and 
legislators who had been in the Council were removed from 
membership. The revised membership had representation from many 
sectors, following the suggestions in the Federal 1202 
legislation. Beginning in the late 1970's, the Council 
gradually lost the confidence of the legislature and of the 
executive staff in the governor's office. It was perceived as 
ineffective in dealing with important issues, and unable to stand 
up to college and university pressures. The sunset revieu" ^•as 
requested by the legislature, and the CPE was placed on the list 
in 1983, for review in 1984, and termination December 31, 19S5 
unless re-authorized • 

A senior auditor spent approximately six months on the 
Council audit, issuing a report (January 21, 1985) noting 
Weaknesses in the agency and recommending its replacement by a 
stronger unit with a new Director. The governor's Office of 
Financial Management (OFM) did not agree to terminate the 
Council, but the Legislative Budget Committer (LBC) agreed. The 
LBC, however, rejected a fvrther recommendation to convert the 
existing coordinating board into a consolidated governing board 
with jurisdiction over all public two year and senior 
institutions . 

Reporting about the same time as the auditor was a three 
year study by a blue ribbon Washington State Temporary Committee 
on Educational Policies, Structure and Management (December IS, 
1984). (See Chapter V.) This group, studying issues at all 
levels of education, recommended replacement of the existing? 
coordinating board by a stronger board of the same type. 

Additionally, discussions about the future of higher 
education and CPE led to creation of tuo special legislative 
study committees during 1984. The study committees produced tv.r 
similar bills, which were negotiated and passed in 1985. This 
created a new Higher Education Coordinating Board effective 
January 1, 1986. The new board was actually the product of a 
total policy process which involved a blue ribbon commission, the 
sunset review, and two legislative study committees. The sunset 
review alone would not have been capable of responding to such 
wide-ranging concern over governance. 

C . Conclusions on Performance Audit and Sunset Review 

First, evaluations of performance are probably an inevitable 
development in an environment demanding increased accountability. 
But it is a difficult task when the agency being evaluated 
operates in a political environment where its policy and 
oversight roles are shared with the legislature, the governor, 
and the institutions. It is difficult to find effective 
indicators of agency performance, and the goals of various groups 
for the agency will vary and may be at odds uith the legal basis 
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for the agency's operations. The ultimate measure of SHEA 
performance is reflected in the way the system of hii^her 
education operates, and some respondents to this inquiry did not 
think that agencies ought to be evaluated apart from the 
evaluation of the system as a whole. At the same time, the SHEA 
has only limited authority (even when it is a governing board) 
over the quality and direction of development of institutions in 
the system; they quite properly have the primary role in 
institutional functioning . 

Second, even if the audit staff represent a 
multi-disciplinary cross-section sensitive to subtle academic 
values, there is the problem of time pressure and audit worklnr.d. 
It is an exceedingly demanding job to move quickly from one 
complex policy area to another, pausing en route often to di^velop 
new measurement indicators of policy outcomes in areas whore the 
internal professionals have long been unable (or sometimes 
unwilling) to reach agreement. 

The performance audit /sunset review process is supposed to 
operate as an accountability process to ensure that agencies are 
operating effectively. But in the states observed, it was 
applied mostly to agencies that were perceived to be in troubl'3, 
so there is a prior judgment about the agency that is put on the 
sunset list. It is not clear that the audits performed a 
formative function of stimulating agency self-improvement in "XT:y 
of the four states we observed. In two cases (Colorado and 
Washington) the agency was eventually terminated and replaced 
with a new one, but in both those states the process of change.* 
involved a citizens' committee as well as the sunset rnvioi:. In 
the two other states, the agency did not view the audit as in any 
way helping them to improve. 

For this process to be constructive in higher education, 
audits x^ill have to look at more than statutory provisions 
regarding the state board's operations. The criteria against 
which the performance of the state agency will be judged need to 
be explicit. To the extent that the agency is judged by the way 
the system of public and private institution functions in meeting 
state needs, it must be recognized that the state agency does not 
have perfect control over the system (even when it is a governing 
board) . It must also be recognized that the state agency cannot 
control the system if it wants effective universities. The 
audit process is not well suited to review of a system that 
depends for its effectiveness on a great deal of autonomy and 
decentralized professional judgment. Most audits thus far have 
been based on the state's obvious interests in efficiency and 
accountabili ty . Whether the audit process can take proper 
account of the decentralized, autonomous nature of higher 
education systems is problematic . 
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The SHEA is better evaluated as a part of a broader 
evaluation of the »-hole state system; the relatively little that 
can be learned from a narrox^^ assessment of state agency 
functioning does not merit the trouble and expense of a 
performance audit/sunset review. 

The complexity and decentralization of the state system, and 
the difficulty of assessing its basic goals of educating 
students, advancing knowledge, and serving the public reqv:ir"-s an 
ongoing assessment process that looks at the broader picture and 
provides periodic feedback for improvements at the operating 
level* This is discussed in moi'O detail in Chapters VI and VTI, 
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Chapter V 



EVALUATION THROUGH SPECIAL STUDIES 
AND BLUE RIBBON COMMISSIONS 



The special study commission is widely used in American 
government at both the federal and state levels to examine 
problems, develop plans and recommendations, and evaluate a wide 
variety of public issues. Higher education has been a favoritf^ 
topic for special studies) in the thirties and forties these 
studies were the predominant way in which higher education was 
evaluated or examined. In the sixties and seventies more states 
developed systematic ongoing planning and evaluation procedures 
through state higher education agencies and state audit agencies, 
but there were still many special studies. The eighties have 
seen an increase in special studies as a resuJt of heightened 
national aj d state interest in the quality of education. In 1933 
Allan Odden, of the Education Commission of the States, 
identified over 200 educational study groups and commissions that 
were formed in the preceding year in the states (Quoted in 
Garland and Hunter, 1987). 

''Blue Ribbon Commissions" can be thought of as a variation 
of special studies. They are defined hy Johnson and Marcus 
(1986, p. iv), as having: a) predetermined length, b) eminent 
individuals fron a variety of backgrounds, c) staff and funds to 
assist in fulfilling their charge, d) a charge to investigate 
and/or recommend changes in structures, functions, origins, cr 
processes." The definition may also include public authorization 
from either the governor, the legislature, or both. Some 
commissions are privately authorized and funded, but they are 
likely to have more di f f iculty in getting their recommendations 
on the public agenda . 

"Blue Ribbon" refers to the caliber of people appointed to 
membership. Most of these commissions are appointed by a public 
official or group, most often the governor, the legislature, or 
some combination of the two. 

Special commissions are usually a part of the incremental 
decision processes of the government, and their work (and their 
impact) can best be understood when placed in the political 
context which led to their establishment. 

Special commissions vary in the kind of recommendations 
made, characteristics of commission members, budgetary support; 
and the defined timetable for the commission. 

A recent study of Blue Ribbon Commissions ( Johnson and 
Marcus, 1986) examined nearly 50 commissions to identify the 



factors that seemed to make some commissions more effective {have 
more impact) than others. Among the factors that identified as 
contributing to success were: (a) having a manageable task, (b) 
having enough time to do the study, (c) adequacy of staff, (d) 
extensive use of outside experts, (e) favorable media relations, 
(f) commissioners who are involved in the implementation process 
(p. iii)* Johnson and Marcus* assessment is that while some 
commissions have had little or no impact, a number of commissions 
have been effective. They conclude that the commission process 
has the potential to lead to positive change and improvement in 
higher education (pp. 72-73). 

A contrary viexv of the value of commissions is presented by 
Paul Peterson (1983) who reviewed the work of six national 
commissions that looked at education (eleirentary and secondary) 
between 1981 and 1983. He says that when judged against the 
usual standards for evaluating policy analysis, these reports 
were disappointing. According to Peterson^ they "... reassert 
what is well known, make exaggerated claims on flimsy evidence, 
pontificate on matters about which there could scarcely be 
agreement ..." (p. 3). Peterson is a critic of the commission 
process itself. He does not tuink special commissions are likely 
to be effective because they are charged to study complex 
problems that are not easily solvable, they have no formal 
power, and they are made up of persons representing diverse 
groups uho are supposed to agree on recommendations in a short 
time period. 

Peterson examined national studies of education rather than 
state studies, which were the primary focus of Johnson and 
Marcus. Mational studies of education cannot limit their 
recommendations to the federal government which is a minor player 
with limited influence on many aspects of education. Instead 
they must find ways to address a diverse group of states and 
institutions th-^t have the responsibility for education. To get 
a message across to a majority of the 50 states and 3,000 higher 
education institutions requires substantial media coverage and 
attention. Reports must be nexvsxvorthy and may need to 
exaggerate and dramatize to make their point. If recommendations 
in the report are too specific, many in the educational audience 
may "tune out" because the report is not seen as applicable to 
the situation in their state or their institution. Education is 
an important item on state agendas, but it is seldom an important 
issue on the national agenda. 

State-level commissions have a more manageable job--they 
can address a few specific audiences who have responsibility- for 
implementation — the governor, the legislature , the state higher 
education agency, the institutions. If the commission is 
appointed by the governor or legislature, it is less dependent on 
the media to get its recommendations to the key decision makers 
in the state. Most states are small enough so that influential 
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persons who are appointed to commissions have ready access to the 
governor and legislators on a personal basis. Clearly, state © 
blue ribbon commissions are more likely to have an impact than 
national ones. 

The problems that state commissions must address vary in 
specificity, but most of them have a broad scope of inquiry. A 
typical charge is: "to study the needs of the state for higher • 
education and the most effective way of providing high quality 
education within the resources available to the state." A 
majority of the commissions have some specific issues or proble.T.s 
to examine within their broad charge; among the most common are 
the governance structure of higher education, finance, and 
institutional mission, role and scope. O 

Governance structure is often on the agenda of a commission 
inquiry. The state agency and the * nsti tutional or system boards 
have vested interests in the continuation of the existing system. 
Therefore, an impartial commission is required to examine the 
structure. The frequent appearance and the controversy over ( 
governance as an issue of special studies is due to the conflict 
between state desires to establish a state "system" of higher 
education and the strong traditions of institutional autonomy. 
There is continuing tension about the amount and type of control 
that the state should exercise, and the appropriate 
accountability and responsibility of each public institution. 9 
(Clark, 1985). This is a complex issue that has no clear 
political solution, and predictably reappears as an issue when 
other problems, such as money shortages or pressure for quality 
improvement arise. Under adverse conditions, special commissions 
tend to look at new governance structures as possible cures or 
solutions to their troubles. i 

To understand the dynamics of special commissions eleven 
special studies that occurred in the seven subject states between 
1979 and 1985 ( each state had at least one ) were examined . See 
Tables One and Two in Chapter I for a summary. Six are 

described in detail because all but one were comprehensive, they i 
all evaluated the performance of both the state agency and the 
state system, and they all made recommendations that could be 
used to evaluate their impact* In each case, a little of the 
political context is provided to enable the reader to understand 
the background of the evaluation. A knowledge of the issues and 
poli tical environment that led to the specii al study is essential i 
to understanding the later impact, or lack thereof, of the 
commission ' s recommendations . 

Colorado 

.Colorado has a complex system of governance. The University 
of Colorado has a constitutional elected board of regents, and 
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there are five other governing boards for higher education in the 
state, Kith the Colorado Commission on Higher Education (CCHE) 
serving as a coordinating board. The legislature is very active 
in oversight of higher education, with many procedural controls 
on higher education that operate primarily through the budget 
process. The governor presents an executive budget, but the 
legislatuia frequei>tly rewrites it; the legislature dominates the 
budget process. 

The CCHE was established in 1965 and was strengthened in 
1970, It was almost abolished in 1975, .when it had its budget 
cut in half, A new director in 1976 rebuilt relationships with 
the legislature, and the commission's influence grew in the late 
1970's and early 1980*s, -With this growing influence came ir.ore 
frequent clashes with the institutions, particularly with the 
University of Colorado, The Colorado legislature has been 
fiscally conservative, and funding increases for higher education 
have been little more (and sometimes less) than inflation in 
most of the years of the past decade. The legislators have been 
concerned with the amount of duplication of programs, with 
competition between institutions for funds, and with the number 
of small institutions outside the major population centers of 
Denver, Colorado Springs and Ft, Collins, 

Colorado was the first state to enact sunset legislation 
(see Chapter IV) and in 1977 the Colorado Commission on Higher 
Education was scheduled for a sunset review in 1982, The sunset 
process which Colorado followed has been described earlier; it 
included a separate review of the agency by the Legislative Audit 
Council and a review by an outside group chosen by the CCHE 
which included former members of the legislature. That outside 
review produced favorable recommendations for continuation of the 
coordinating board, but it also included a recommendation to the 
legislature that the governance arrangements in the state be 
studied and, if possible, simplified. The independent review by 
the legislative auditor recommended the continuation of the CCHE, 
but also suggested that the legislature reviexv- the governance 
arrangements in higher education with the objective of making 
them more efficient and accountable. 

At about the same time, a downturn in oil prices affected 
Colorado's revenues and heightened the legislators' interest in 
achieving a more economical operation of the higher education 
system. Many political leaders and some educational leaders 
believed that changes were needed, but there was no consensus on 
what those changes should be. 

Representative Schauer was a member of the Legislative Audit 
Committee, and he became interested in the issue of governance in 
Colorado, In 1984, he introduced a bill to reorganize higher 
education under a single governing board. There were extensive 
hearings on the topic, and it became clear that there were other 
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problems to consider in addition to governance. It was also 
obvious that there was not much support among the educators for 
the notion of a single governing board. The bill could not pass, 
so the legislature authorized an interim citizens' study 
committee to examine the Colorado higher education system paying 
particular atten tion to ways that a streamlined governance 
structure might improve the use of resources and advance the 
quality of Colorado's higher education system. 

The members of the committee included Representative Schauer 
and eight other citizens who were drawn from different areas of 
the state. Three were appointed by the senate, three by the 
house, and three by the governor. They had seven months to 
complete their work and report to the next session of the 
legislature. The chairman was acknowledged by nearly all 
respondents as an outstanding choice who shaped the work of the 
commission in a positive direction. The committee had an 
experienced staff. It was drawn in part from the staff of the 
CCHE and in part from the legislative audit staff; limited use 
was made of consultants from the Education Commission of the 
States. The CCHE endorsed the importance of tht Citizens Study 
Committee, and indicated a willingness to go out of business if 
the study committee felt that would help implement a better 
pattern for governing higher education. Thus, CCHE's disinterest 
in fighting a reorganization or governance change greatly reduced* 
the political problems faced by the Study Commission The 
commission held a number of public hearings around the stale, and 
issued in January of 1985 a sixty-page report which dealt broadly 
with the future of higher education in the state. 

The commission adopted four major goals which they sought to 
fulfill through their recommendations. 

Our recommendations are driven by educational 
and management judgments rather than by what 
might be politically most feasible. We have 
no doubt that implementation of our 
recommendations will substantially improve 
the current situation. We urge that our 
recommendations and all competing options be 
evaluated against the four major goals. 

First, the proposal must clarify the purposes of educationa.1 
institutions through differentiated roles and missions. 

Secondly, the proposal must provide responsibility for 
developing statewide policies linked with the necessary authority 
and the accountability for implementing those policies. 

Third, the proposal must enable and encourage decentralized 
management of education by providing a decisive process for 
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resolving conflict based on sound educational and management 
criteria* 

Fourth, the proposal must provide mutuality in the 
commitment by the educational community to quality, access, 
diversity, efficiency, and accountability in exchange for 
commitment by the citizens of Colorado to adequate funding," 

The commiss ion organized its recommendations into tho^e 
directed to the legislature, and those directed to the governing 
authority. The legislative recommendations called for creation 
of a single governing board with management boards for each 
campus , legislative definition of the broad role and mission cf 
each institution, and revision of the funding formula to provide 
flexibility and to reduce the emphasis on student enrollment as 
tha basis for funding. 

The recommendations directed to the governing authority 
dealt with improving the quality of higher education, providing 
management flexibility and increased accountability at the 
institutional level, and assuring good communication and 
institutional participation in the development of state education 
policy ♦ 

The recommendations were included in a new bill introduced 
by Representative Schauer in 1985* The subsequent legislative 
hearings made it was clear there still was not a consensus for a, 
single board* The bill was modified to re-establish a new, 
strengthened CCHE with increased authority to terminate existing 
programs ♦ Most of the other recommendations of the Citizens 
Committee were included in the legislative mandate to the new 
CCHE, while making it clear that CCHE was a policy board that 
should not get involved in institutional management* 

A fairly high proportion of the recommendations of the study 
commission were adopted in the legislation for the stronger 
coordinating board. There were several factors responsible for 
this* The chairman and the special commission members worked 
hard at understanding the position of major groups on the key 
issues in higher education* Secondly, they had a senior, 
experienced staff who knew the issues, what information was 
available, and who could facilitate the work of the special study 
commission* Third, there was a feeling on the part of 
legislative leadership that it Xv-as time for a change* The 
governance , role and mission duplication, and financing were 
issues that had been part of the political and educational 
agenda in Colorado for several years, and people wanted to settle 
them and move ahead* A fourth reason that the commission's 
recommendations were influential was that the CCHE itself 
supported the idea of change and was willing to be abolished if 
that was in the best interest of the state. The executive 
director of CCHE had left the state before the study began. The 
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president of the University of Colorado, who had opposed some 
CCHE proposals had also resigned. There was a clean slate in 
terms of much of the leadership in the state. At the time that 
the new CCHE began its work under its new legislative mandate, 
there were no public four-year university or college presidents 
in the state who had been in their position for more than three 
years ♦ 

To summarize, the political and educational environment ^:as 
right for a change. None of the key groups fought changes except 
for the idea of a single governing board. The issues before the 
study commission were central policy questions about the future 
direction an^; role and mission of institutions. The issues had 
been extensively debated and people were ready to resolve them 
and move on. The outcome of this particular study commission was 
positive for the state and a citizens' commission was the right 
process to address these issues given the history and political 
environment in the state. 



Florida 

Florida is a state with ambitions to be in the top 
nationally in most areas, including higher education. It is also 
a state with no income tax, and as a result it has below-average 
tax capacity and tax effort which provides limited public 
resources with which to fulfill its ambitions. 

This background is important in understanding the events 
that led up to a 1979 study by a joint legislative and 
gubernatorial blue ribbon commission which reported a set of 
recommendations to the 1980 Legislature . 

The special commission was generated by discussions about 
the role and mission of the new universities in south and centr'^1 
Florida that were built in the 60 's and early 70 *s. The older 
established University of Florida and Florida State are in the 
northern and less populated part of the state. They faced 
increasing competition from the four newer universities in 
central and south Florida for already inadequate tax revenues. In 
addition there was the concern that undergraduate education 
wasn't as good as it needed to be. To make Florida a national 
leader, there needed to be more attention on quality improvement 
at all of the institutions. 

Florida state government has more sectional politics than 
many states ; the many newcomers to the state have more allegiance 
to their local area than they do to the state as a whole. It is 
also a large state with a substantial investment ( in the last 20 
years) in planning, evaluation, and modern "rational" management 
procedures in higher education. It has been a leader among the 
states in building a sizeable professional staff for its 
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legislature, and in -^.ttracting professionals to executive branch 
agencies . 

Sectional politics and comprehensive statewide planning and 
management make a contentious mixture, and this has been most 
evident in higher education. The Board of Regents, as the 
governing board for the nine senior universities, de^'eloped a 
plan in 1970 for growth and development of higher education. 
This 'plan did not satisfy the four new central and south Florida 
universities or their legislative supporters. Agreement was not 
possible on clear role and mission delineations. Two of the nei: 
univer.sitie? were upper division only (i.e. they had no freshmen 
or sophomores); they grew much less than had been projected, and 
the universities and their legislators were now pushing for 
broador missions and the addition of freshmen and sophomore 
students . 

Feeling that the Board of Regents was biased in favor of the 
older established universities, legislators from south Florida 
proposed legislation in 1978 to abolish the Board of Regents and 
set up local boards with the idea that each institution might do 
better with a locally guided development effort. The governor 
and some of the state leaders were opposed to this idea, and the 
proposed legislation to abolish the Board was sent to a Joint 
Legislative-Executive Study Commission. The twenty-one member 
commission was made up of a combination of citizens and 
legislators, many of whom had strong opinions and well-formed 
interests in one or another possible solutions. The commission 
staff was from the governor's office, the legislative staff, and 
the Department of Education staff. In addition, it contracted 
with the Academy for Educational Development to do a study and 
develop recommendations about several major issi^es . The 
Commission was charged to look at what the state needed in higher 
education , institutional roles and missions , funding and funding 
procedures, how quality of the system might be enhanced, as well 
as governance. All this x>;as to be done in a six-month period. 

The outside consultant group, the Academy for Educational 
Development (AED) , developed a report that dealt Xvith the 
questions facing the Joint Executive-Legislative Commission: 
"State Needs and Institutional Role and Scope," "Financing 
Postsecondary Education in Florida," "Quality and How It Can Be 
Improved," and finally "The Governance of Postsecondary 
Education." This last part of the report by the Academy 
consultants was the most controversial part . The Academy ' s 
recommendations reflected the influence of state politics, 
especially about whether there should be individual boards and 
x>7hether the Board of Regents should be abolished. Some of the 
members of the Commiss ion had expected the AED report to include 
a recommendation for local board governance, but it did not. 
Instead it outlined the strengths and weaknesses of various 
approaches and left the Commission the job of resolution. 
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Because the Academy study was believed to be influenced 
po"itically, it lost credibility, particularly wiLn some of the 
key professionals who staffed the Executive-Legislative 
Commission. They felt that the role of the outside consultants 
was to lay out the facts and recommendations objectively, and not 
respond to political pressures. The Executive-Legislative 
Commission issued its own report a^out a month later. It 
recommended the continuation of the Board of Regents, but also 
the establishment of a new Postsecondary Planning Commission that 
would assist the constitutional State Board of Education. In 
Florida, the Cabinet (which is the governor, and six other 
elected state officials) serves as the State Board of Education, 
responsible for all levels of education including the 
universities, community colleges, and K-12. Recommendations fcr 
financing and for the future role and mission of each 
institution were also put forward by the study commission. 

Implementation of the recommendations was impacted by 
financial considerations. In Florida, the state has allocated 
between ten and eleven percent of total state tax revenues to 
higher education during most of the preceding two decades. These 
funds have grown at about the rate of the overall state economy, 
but since Florida has no personal income tax, tax revenues 
reflect a relatively low total tax effort. Meanwhile, the fact cf 
slow growth of new revenue meant that if the new universities in 
south Florida got more of the pie, the existing universities in 
north Florida would get less. The recommendations of the 
commission did not alter this basic funding predicament, and the 
recommendation to move from an enrollment-based funding to a 
program-based funding has been only partially implemented. At 
the same time, the report responded to the south Florida 
pressures by indicating that there should be a plan for a 
comprehensive doctoral level university in the southeastern part 
of Florida. This was to be accomplished hy combining the efforts 
of two public universities x>7ith the University of Miami. Seven 
years later, this plan is now being implemented 
with supplemental funding for the t\<o public universities, 
although the funding level for the total system remains at nearly 
the same percentage level . 

If Florida were to provide adequate funds, it could have 
five or six comprehensive , nationally- recognized universities , a 
top-notch community college system , and first class public 
schools. But given the funding constraints and the political 
strength of the central and south Florida areas, the grp between 
aspirations and resources will continue . 

The commission's recommendations did lead to the 
establishment of a new Postsecondary Education Planning 
Commission (PEPC) to serve the Cabinet Board of Education. PEPC 
has had considerable success in getting the important policy 
problems on the agenda of the governor and legislature, and 
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progress has been made in some areas including clarification 
(seven years after the commission's recommendations) of the role 
and missions of the southeast Florida institutions. Higher 
education's relative funding position in the state budget has not 
improved . 

The Florida experience suggests that citizen commit t*^r^s mp.v 
be ineffective in making recommendations about funding, becausfj 
funding char.ges require new political commitments on the part of 
the public, legislators and governors. Citizen committees can 
bring fiscal issues into focus in the public debate, but it is 
not realistic in most situations to expect much change from blue 
ribbon commission recommendations about financing. 

The Florida studj^ made extensive use of outside experts; the 
Commission not only had a major contract with the Academy for 
Educational Development, it also brought in other consultants to 
critique the AED recommendations. Respondents did not rate the 
contribution of the outside consultants very highly. The 
problems the Commission faced were not technical, but 
political--how much is the state willing to spend on higher 
education, and how local aspirations for higher education in 
politically influential metropolitan areas can be satisfied. 
With these kinds of problems, the potential for outside 
consultants to make much of a contribution is limited. In the - 
judgment of the respondents, the Joint Legislative and E::ecvitive 
Commission was B^t least partiallj" successful in suggesting a new 
way to work on the problems. 



Washington 

Of all the special studies examined, a special study of 
education in Washington had the moao ^../niprehensive charge (to 
look at all levels of education), took tre longest time ( hwo and 
one-half years), and cost the most monej'' (over a million 
dolla.?s — most of it from private sources). If bigger is better, 
this should be the most influential of the studies presented. 

This study began when the state of Washington had ueen in 
an economic recession for several years . Washington had a 
tradition of progressive state government and a lot of citizen 
participation, but recent economic problems had increased fiscal 
conservati veness . Consultation and participation are emphasized 
in Washington and special studj' commissions are common. The 
legislature has also developed a substantial professional staff 
who tend to be non-partisan, and often move between executive and 
legislativ^e employment. Olympia, as more than one respondent 
pointed out, is a small town, government is still conducted on a 
personal basis, and people (and agencies) develop reputations 
which strongly influence their effectiveness with other parts of 
state government . 
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Higher education has been a budget priority in Washington, 
and as a result, the University of Washington has developed as 
one of the best universities in the nation. A large community 
college system contributed to a relatively high college 
attendance rate. Statewide coordination of higher education is a 
recent activity. A strong tradition of institutional autonomy 
exists and a coordinating board was established primarily as n 
planning agency with only advisory powers. There was also a 
pattern of active legislative involvement in educational policy- 
making in higher education. About half of the initial 
coordinating board were legislators or institutional 
representatives. It was a large and somewhat unwieldy body. Its 
early and limited success is attributed to the skill of its ffrst 
director (Jim Furman), to the caliber of the initial board 
appointees , and to the support of the governor and the 
legislature. In 1975, the coordinating board was reorganized as 
the Council on Postsecondary Education (CPE), and most 
institutional members and all the legislative members were 
removed, making it primarily a citizen board. The CPE had a 
limited role in budgeting and was primarily a planning board that 
had to rely on persuasion rather than regulatory power to 
resolve program conflicts between institutions. 

The downturn in the Washington economy in the late 70 's made 
it increasingly difficult to deal with problems. A succession 
one-term governors contributed to political uncertainty about 
state higher educational policy-making. The legislature was 
interested in having higher education operate in a more 
efficient manner and was frustrated with the CPE because it 
seemed unable to resolve turf conflicts between institutions. As 
the 1980 's began, these pressures and cone ns about the 
effectiveness of higher educat.»on led to several simultaneous 
evaluations and recommendations. 

Some members of the Washington legislature were concerned 
that higher education and elementary/secondary education werr» 
competing for scarce state dollars , and there was no objective 
overview or a road map for education. These legislators wanted a 
comprehensive study that focus ^d on goals , roles of various 
institutions and sectors, and program financing". They authorized 
a Temporary Committee on Educational Policies, Structure, and 
Management in 1982. Initially, it was perceived that much of the 
study would focus on higher education , but at the time it was 
authorized the national interest was shi f t ing very sharply toward 
the problems of effective . education in the elementary/secondary 
schools. The Temporary Committee wound up using a majority of 
its time and effort on elementary/secondary education. The 
Temporary Committee also examined teacher education and did 
address some questions of governance and finance and program 
quality in higher education. However, the majority of its 53 
recommendations dealt with elementary/secondary matters and the 
problems of improving the public schools. 
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The committee had 17 members--13 appointed by the govornor 
and four legislative members (one Democrat and Republican from 
the Senate and one Democrat and Republican from the House). Most 
of the funding for the study was raised externally. The initicil 
legislative appropriation was for $25,000 with an additional 
$100,000 matching any private contributions. The staff director, 
Dr. Will iam Chance, had been in state government in Washington 
for more than a decade with the CPE and was knowledgeable abcut 
state higher education operations. 

The Temporary Committee spent a good deal of its first 
year's existence collecting information and arranging for some 
external studies and obtaining the additional funds necessary to 
carry out a comprehensive study. It made an interim report in 
1984 and a final report in January of 1985. The charge to the 
Temporary Committee was broad: to "investigate thoroughly the 
entire educational complex in Washington state.*' In addition, 
they were asked to examine the following specific issues: 

coordination, needs of students and response 
to those needs, the role and missions of the 
components , educational diversity and 
independence , obstacles to orderly student 
progression , open access , efficiency, 
duplication , accreditation, graduation and 
entrance requirements f rom high school to 
postsecondary, efficient uses of the public 
dollars , ways to improve the system possibly 
through managerial reorganization or 
combining of components , accountability of 
the various levels , student achievement , and 
a determination of what constitutes good 
ins truct ion . 

With this broad and detailed charge, it is not surprising that 
the commission's work took two and one-half years to complete. 

Most of the people who evaluated the work of the Temporary 
Committee felt that it had too broad a charge and, as a result, 
made too many wide-ranging recommendations. Respondents thought 
it had some impact on elementary/secondary education, but 
limited influence on higher education. 

There is an important problem in assessing the Temporary 
Commission's ijnpact. There were three other studies under way at 
the same time as the Temporary Committee, each dealing with 
higher education governance.-; lin^nce, and the roles and missions 
of institutions. The first ox these was a sunset review of the 
Council on Postsecondary Education (CPE) which is described in 
the previous chapter. The sunset report was issued at about the 
same time that the Temporary Committee report was issued, and 
there were similarities in the way the two reports evaluated 
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governance arrangements. The Temporary Committee had sponsored a 
special study of governance of higher education by Lyman Glenny 
and Frank Box^7en of the University of California. This study 
pointed out x^7eaknesses in the existing pattern of operation of 
the CPE, and made a number of suggestions about Kays that it 
could be strengthened as a policy board. These ideas \:eve 
present in some of the other reports, although each study made 
its own recommendations in its oxy-n termor. 

In addition to the Temporary Committee and sunset review, 
two other studies Xv-ere done by special Senate and House 
committees xvhich included lay members. The Senate Committee xcas 
organized and chaired by Senator Nita Rinehart, uho had the 
University of Washington in her Seattle district. Senator 
Rinehart was also a member of the committee that provides 
oversight to the Legislative Audit Council, so she K'as familiar 
uith the sunset review of the Council on Postsecondary Education. 
The major purpose of her committee uas to make a new 
recommendation for governance if the sunset audit put the CPE out 
of business. Senator Rinehart wanted the recommendations to be 
acceptable to higher education , and therefore involved 
representatives from the colleges and universities extensi\ely in 
hearings and in developing recommendations. Higher education 
leaders x^ere uneasy about the possibility of a single "super 
board" an idea Xv-hich was appealing to some members of the 
legislature . 

The House Committee x^as organized and headed by 
Representative Helen Sommers, and it also developed a governance 
proposal to substitute for the CPE if it uere abolished. Each 
committee also examined other issues, but the question of 
governance Xvas the primary topic. In describing their 
committees, both Rinehart and Sommers identified them as 
legislative committees ; they involved educators to meet the 
Washington traditions of citizen participation. They used the 
regular committee staff of the Senate and House Education 
Committees as staff to the special studies. The Temporary 
Committee Xv-as described as a ci tizens committee , even though i t 
had four legislative members. The difference x^jas in the control 
of the recommendations . 

So, there were four different study and evaluation groups 
looking at the governance issue. The Temporary Committee had 
recommendations from its outside consultants; the others relied 
on public hearings and staf f work to develop their 
recommendations . The consultants ' report to the Temporary 
Committee was the first recommendation to be made public ; that 
being a call for a strengthened coordination and state higher 
education policy process . The sunset reviex,? recommended the 
termination of the old board and the creation of a more effective 
coordinating mechanism without specifying in detail what thar 
might be. The two legislative committees produced si.milar 
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recommendations about a neu coordinating board \:ith close" ties 
to the legislative process. The legislative committees' 
recommendations were successfully combined into a single bill 
which scheduled the old board to be phased out January 1, 1986, 
and a new board to begin work the same day. 

It is possible to identify similarities between thvt final 
outcome and the recommendations of the Temporary Committee, b;it 
participants in the process give different assessments of the 
actual impact which the Temporary Committee had on the final 
legislative action. In part, that is because legislators like t 
take the credit for whatever legislation is passed, and whil^^ 
they may borrow freely from ideas that are presented in other 
reports, they necessarily want to identify them as their orn, 
particularly if thet bill is seen as a good one after it passes. 
Most neutral respondents, those who xcere not associated with 
either the Temporary Committee or the legislative comrl t te*i'i-^ , 
f el t the ci tizen committee had modest influence and that son ? o f 
the ideas that it presented were incorporated into the final 
legislation; however they also believe the final legislation 
might have looked about the same even without the work of the 
temporary committee. The legislators weie engaged in a typi?^' 
incremental-political process, finding a consensus that 
"satisfices" the major interest groups. 

As one informant put it to us, an important role of citi.'zen 
committees is to get new ideas on the agenda. For those id»^n^: to 
go anywhere, they must be picked up by legislators who make t h^:n 
their own and x^-'ho then can claim credit for whatever outcom^^ 
occurs. Thus, it is difficult to trace the impact of tht? 
Temporary Committee in Washington, in part because it producs-d *i 
very broad report with many recommendations, and in part bhfcau^:o 
there were so many other groups involved in considering the snmt=* 
issue in the same time period. Several respondents f eJ t that Ih 
Temporary Committee had too broad a charter from the legisl.^tur^ 
Also contributing to its limited impact \cas the length of time 
taken to produce its report and recommendations. The governor 
who had appointed it had gone out of office and the new goverrr^r 
had just entered office when the report came out in 1985, and th 
new governor felt no sense of ownership. 

A 1986 legislative assessment of the impact of the 
recommendations from the Temporary Committee showed that only 
about ten percent of the fifty recommendations resulted in 
legislation of a positive sort. This, of course, is just a 
first-year effect. It x^as more revealing that this analysis 
indicated that the legislature did not have any serious 
discussion of about half of the recommendations. It seems 13 k^ 
that, as one legislator put it, the Temporary Committee had an 
overload of recommendations. He felt that the legislature could 
only focus on a few issues at a time and a comprehensive reform 
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package was not something that the Washington legislature was 
prepared to deal with at the time that the report came out. 

The impact of the Temporary Committee also should be 
assessed in relation to the political environment: a recession 
with high unemployment for several years; a succession of 
one-term governors; and considerable change in legislative 
leadership. There was no clear political leadership ready to 
take up a comprehensive education agenda. 

Given all this, the Temporary Committee's impact may have 
be en about as great as it is reasonable to expect. The committee 
members were influential and had a competent staff, although its 
political connections were seen as both assets and liabilities. 
It had some impact on the thinking of leadership in the 
state — but more on elementary/secondary education than on higher 
education. The environment was not right for a major change in 
higher education, and most people assess the committee's role as 
one of background rather than proximate influence in the changes 
that finally took place in the state in 1985 and 1986. 

Idaho 

In 1978 Idaho's version of property tax limitation. 
Proposition One, known as the One Percent Initiative, was 
approved. As a consequence, the local taxes available fo^- 
elementary/secondary education declined, and significantly more 
state resources had to be diverted to the public schools. The 
higher education portion of the state ' s general fund 
appropriation dropped from 22% to almost 15% in five years. 
Higher education suffered even more when an economic downturn hit 
the state. Between 1980 and 1984 Idaho was next to the bottom 
among the states in the growth of state and local appropriations. 
Constant dollar appropriations per r.tudent declined more than 
21%. 

By the early 1980 's there was concern that the Idaho State 
Board of Education . which governs the four publ ic senior 
institutions, might be jeopardizing the quality of higher 
education by trying to conduct business as usual in the face of 
declining resources • 

A group of prominent citizens in the Idaho Association of 
Commerce and Industry (lACI) obtained private funding to form a 
study commission. The lACI appointed 35 members including four 
legislators, several college presidents, and many prestigious 
businessmen. A fifteen-person advisory group included a 
representative from the governor's office and the current and 
preceding executive directors of higher education of the State 
Board of Education. The study group then hired a team of 
consultants from the Education Commission of the States (ECS) to 
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undertake the staff work and to prepare its ox>n set of 
recommendations. The ECS Report appeared in March 1983. 

After holding eight public meetings with over 370 citizens 
appearing to testify, lACI issued its own report in November 
1983. It largely agreed with the ECS report. The 
recommendations were to separate a strong Board for Higher 
Education from the State Board of Education, to expand the 
v-.ommunity colleges beyond the tvo already in existence, and 
finally, to generate more state money for higher faculty 
salari es, better bui Iding maintenance, and a Leeds-^based stud.ent 
aid program to compensate for charging students higher tuition. 
Higher admission standards and better definition of institutional 
role and mission were also recommended. 

The lACI Report got caught in a political cross-fire as 
legislators introduced a bill to raise business taxes as a way of 
funding the recommendations for higher education. The lACI then 
lobbied to defeat the tax proposal while the lACI study group was 
urging the state to spend more money on higher education. 

Also, the report made a number of recommendations for major 
changes, and these drew active opposition from several groups, 
each attacking a different recommendation. Eventually,, the 
objecting groups stopped any significant progress toward 
implementation of the I.ACI recommendations. 

The membership mixture was judged by most respondents to 
have been adequate, but several commented on other possible 
weaknesses in the process: 

1 . The in-state legislative members were seen to be 
fa\'orably biased toward education before the study and therefore 
lacked credibility with fellow legislators who were critical of 
higher education* 

2. There was some mild resentment of out-of-state experts 
and their criticism of Idaho's system of higher education. 

3. Even though there was a member of the governor's office 
on the advisory group, several respondents reported the 
governor's office did not work hard for the changes. 
Incidentally, Idaho several times had a strong, conservative. 
Republican majority in both houses and a Democratic governor; not 
a formula destined to produce much legislation or to let the 
governor operate independently • 

Idaho interviewees believe that the bedrock problems were 
political rather than educational. Respondents emphasized the 
degree of regional factionalism in Idaho: north Idaho tends to 
be Democratic and to protect Lewis and Clark State College; 
southwest Idaho has population and economic power that is 
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bitterly resented by the rest of the state; and southeast Idaho 
is a Mormon stronghold that often looks more to Utah than to 
Idaho for higher education services and therefore is politically 
reluctant to support increased costs for higher education. 

Furthermore, elementarjVsecondary education is u'ell 
organized and influential in Idaho. The elected superintendent 
of public instruction ensures that elementary /secondary 
education gets its "share" of any tax increase. (The present 
superintendent, it must be said, has not used his voting 
membership on the State Board of Education to force fund'ng 
issues into an "ei ther/or " elementary/ secondary or higher 
education context.) Idaho is a state with a very small 
population and the number of people who are influential in higher 
education constitutes a small network. A few individuals ca.n 
block action by their opposition. 

The lACI report is a good example of the difficult:* of 
implementing blue ribbon commission recommendations for funding 
changes. Since the lACI Commission was not appointed by either 
the governor or legislature, they were not committed to work for 
the recommendations. In addition, when the lACI as an 
organization opposed the business tax increase, this neutralized 
the interest group support for the recommendations. 

Although not many of the lACI recommendations XN'ere 
implemented, several observers urged caution in declaring the 
mixed blue ribbon/outside consultant process a failure. They 
felt that one needed a longer timetable in evaluating reports 
that appeared in 1983, for the policy process in Idaho moVes to a 
s] .ower beat. As evidence, they cite that both the Northwest 
Regional Accreditation Report (described earlier in Chapter III) 
and a 1986 legislative committee report on higher education 
appeared to build on lACI recommendations. So the seeds fcr 
useful policy changes may have been planted and may yet produce 
fruit • 



Outside Consultant Reviews 

The final two state studies to be examined in detail were not 
"blue ribbon" study commissions. Each study was done by an 
out-of-state consultant group. In the case of West Virginia, 
the consultant group reported to and was hired by the 
legislature. Some members of the legislature had been unhappy 
with the role of the Board of Regents, and wanted an examination 
of the governance issues. This was the most narrowly focused of 
the study examples included in this report. 

In South Carolina, the outside consultant group was hired by 
the Commission on Higher Education to do a comprehensive study of 
the system. The South Carolina legislature funded the study, and 
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also put some constraints on the scope of the studv (it was not 
to consider any mergers of institutions) • The basic mandate ^or 
the study came from the 3.C. Coipmission on Higher Education, 
which was serving as its own "blue ribbon" commission. 



West Virginia 

The West Virginia Board of Regents is a consolidated 
governing board created in 1969. The board had several clashes 
with both the state government and the institutions during its 
early years. Governor Moore, who had appointed the original 
Regents, became critical of the group when it opposed his 
proposals to create a second medical school at Marshall 
University and to incorporate the private Greenbrier School of 
Osteopathy as a third medical school in the state system. The 
governor's wishes prevailed, but not until a divisive 
disagreement had occurred 

At an early point in the new boari^s existence, the 
chancellor of the new board sent out the word that institutions 
(particularly West Virginia University),, were no longer to engag3 
in "end-runs" around the board by lobbying for more 
appropriations. Presidents who had previously enjoyed direct 
access to political decision-makers did not welcome such 
constraints. The remainder of the 1970'i found the state beard 
struggling with both substantive agendas and procedural relations 
with its constituents. 

Although the Board had published two stat^ plans in 1972 and 
1979, had created a set of community colleges, and had provided 
funds to restore the accreditation standing of several public 
colleges with library problems, key legislators felt that the 
Board ' s ten -year anniversary in 1979 was a good time to evaluat?^ 
its effectiveness and therefore hired the .Academy for 
Educational Development (AED) to undertake this task. 

The AED report's major recommendation was to abolish the 
Board of Regents and create in its place a state coordinat ing 
board, with governing boards for each of the three public 
segments: the university system, the state colleges, and the 
community colleges. This recommendation was based partlj* on a 
lengthy section of the report analyzing (without particular 
reference to West Virginia) the general strengths of state 
coordinating boards as compared to state governing boards, and 
partly on a set of criticisms of the operating style and 
functions of the existing Board of Regents. 

In addition, the AED report provided 23 recommendations tc 
improve the Board's operations in case the legislature chose to 
reject the primary option and keep the Regents in place. The 
Board itself urged the rejection of the primary recommendation 
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but indicated that it found no trouble with the 23 improvement 
ideas . 

In 1981, a law passed strengthening the Board of Regents and 
implementing several of the AED recommendations--in particular, 
that the Institutional Advisory Boards be made statutory and 
given more delegated powers, mandating the program wValuafcion 
process by which all existing programs would be ':jvaluated every 
five years, and mandating an explicit master planning process 
with broad participation by faculty, students and professional 
staff. As a consequence, the Board published its third state 
plan. Agenda for Action, in 1985 and it seemed to involve 
widespread participation as recommended. In 1982, the Regents 
published Policy Bulletin No. 11 which required the evaluation of 
all programs every five years. Policy Bulletin No. 59 in 198 1 
implemented the AED recommendation to evaluate campus presidents 
at least once every four years. 

The present Board of Regents is involved in planning, 
program evaluation, and presidential evaluation along the lines 
that the AED recommended and those which were put into the '81 
law* To that extent the out-of-state report was moderately 
successful in improving the Board of Regents despite the fact 
that its primary Option 1 was rejected. 

But the evaluation of the AED role cannot stop there, for as 
mentioned earl ier in Chapter III, the West Virginia performance 
audit staff relied heavily on the AED report and subsequent Board 
reactions when the performance audit report was undertakeii in 
1984. The influence of the report could be seen five years 
later.' Again, it is clear that the assessment of the impact cf 
most special studies should be generous in its time frame. 



South Carolina 

South Carolina is a relatively small state with a comple:: 
system of higher education. There are two comprehensive 
universities, a medical university (plus a second medical school 
at the University of South Carolina) , several state colleges, 
several two- and four-year branch campuses of the University of 
South Carolina, plus a system of two-year vocational schools. 
The system has added a private institation and a municipal 
college to the state system in recent years . There are a number 
of private institutions that benefit from a state tuition 
equalization grant program. The system is coordinated by the 
South Carolina Commission on Higher Education (CHE) x%hich has 
limited regulatory powers . 

The South Carolina economy has been growing, and the state 
has supported its higher education system relatively well 
compared to other states with similar levels of income. But 
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there is also a feeling among some state leaders that the 
available dollars have been spread among too many institutions, 
and that there is excessive program duplication. 

In 1985 the South Carolina Commission on Higher Education 
(CHE) persuaded the legislature to appropriate $350,000 for a 
comprehensive evaluation of higher education, to be done by 
consultants f rom outs ide the state • This was not a blue ribbon 
commission study. The CHE was the key group that received the 
consultants' report, and the CHE was responsible for presenting 
the report to the legislature, governor, and the public, and for 
whatever implementation occurred. 

The study was designed to conduct a comprehensive broad- 
based review of higher education in the state and to project 
needs for the next decade. The CHE thus got an objective lock 
from an outside group of experts, but as the citizen group 
responsible for the study, CHE brought its existing assets and 
liabilities to bear on the job of getting support for the 
recommendations . 

The study was planned because the Commission wanted to seize 
the initiative and present a comprehensive plan for improvement. 
State government in South Carolina is conducted primarily on a 
personal basis, and sectional interests are strong. There is a 
small professional staff for the legislature, and education has a 
strong tradition of institutional autonomy from state oversight. 
Legislators not only appoint most of the higher education board 
members, but also serve on several higher education campus 
boards. The CHE was established in 1967 and initially had 
institutional representatives on the Board who were removed in 
the 1978 reorganization. Currently one-third of the 18 CHE 
members are appointed by the governor from nominees of the 
legislators f rom each congressional district. The other 
two-thirds are appointed by the governor with the advice and 
consent of the legislators from the appointee's congressional 
district. This appointment process keeps local interests at the 
forefront, and makes it hard to develop a statewide perspective. 

The CHE has had a mixed record of influence on the 
development of higher education policy, CHE's efforts to prevent 
program duplication in expensive areas like medicine and 
engineering were unsuccessful, and the Commission was subjected 
to frequent end runs to the legislature by institutions when the 
CHE tried to reduce program duplication. Some legislators think 
higher education is over-built and has too much program 
duplication, but local interests have been able to block CHE 
efforts to deal with duplication, especially in health education 
and two-year occupational education. 

Gc/ernor Riley, who served from 1978 to 1986, was a strong 
advocate for improved schools and for education reform, and had 
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Kon a tax increase for a comprehensive public school reform 
program in 1983, Higher education did not benefit much from the 
increased tax revenues; the level of support for South Carolina 
higher education among the southern states was better than the 
South Carolina support for the public schools, and the governor 
and legislature felt it was the public schools' turn, 

Fred Sheheen was the key figure in getting the outside 
evaluation funded and organized. He became chairman of the CHE 
in 1983, and served as chair until his term ended in 1986. He 
had served on the Commission in the early 1970' s and was 
appointed again by Gov-ernor Riley in 1979, Sheheen's brother 
was a key member of the House and is currently the Speaker cf the 
House Sheheen was a supporter of Riley's elementary/secondary 
education initiatives, and he felt that a companion package of 
comprehensive higher education improvements usls needed to improve 
the quality of higher education and complement the work that had 
been done at elementary/secondary level. He saw the study as a 
necessary basis for the Commission's development of a higher 
education reform package, 

Sheheen was personally responsible for working with the 
legislature and the governor to get the $350,000 from the 
legislature for the study which was to look at several issues he 
wanted the study to examine: program duplication; the 
possibility for a comprehensive community col lege system 
combining the two-year branches of the University of South 
Carolina, with the technical schools (this part of the study was 
eliminated by the legislature when it appropriated the funds); 
quality enhancement 'recommendations that would parallel those in 
the elementary/secondary package; the role of university 
research in economic development, and finally, how the commission 
itself could become a more effective agency, 

Sheheen and the Commission selected the Augenblick, Van de 
Water and Associates (AVA) from Denver, Colorado, to conduct the 
outside study, and they, in turn, selected about twenty other 
experts to come in and look at different facets of the state 
higher education system, (Both authors of this report were 
involved as outside consultants in this study,) The study 
commission staff operated much like the staff of a citizens' blue 
ribbon committee. They interviewed a large number of people in 
higher education and in the executive and legislative branches of 
state government , They examined several issues , commissioned a 
number of outside papers, and in six months (February 1986) 
produced a 110-page report with twenty-two recommendations to the 
Commission, Twelve recommendations were directed at 
strengthening the CHE, one of which would require additional 
legislation to increase the responsibility and authority of the 
Commission in the area of program elimination , Ten 
recommendations addressed ways of improving the quality of the 
educational programs and services in the state, 
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The Commission on Higher Education set up a process for 
implementing the recommendations and for presenting selected 
recommendations to the legislature. The report did not come out 
at a good time for implementation. Chairman Sheheen, who had 
been the principal advocate of the study process, was ending his 
term on the Commission, The Executive Director of the CHE for 
the past twelve years, Howard Boozer, had announced his 
retirement to be effective a few months after the report was 
received. Members of the legislature were in the middle of their 
final session before the 1986 election. The state was facing a 
downturn in revenue which was threatening the full funding of the 
reform packages in elementary/secondary education that had been 
passed two years previously. Governor Riley was in his last 
year as governor, and could not run to succeed himself. Xc thins* 
was done with the recommendations by the 1986 legislature. 

The CHE appointed six task forces of college and university 
people, laymen, legislators, and commission staff to review the 
work of the consultants and make further recommendations aimed at 
reaching the 1987 legislative session. The study consultants had 
interviewed a number of college and university teachers and 
derived some suggestions from them. This had r. t built an 
adequate sense of ownership of the study for institutional 
leaders; and the six task force studies were designed in part tc 
gain inrti tutional support for the recommendations. The CHE 
used the six task forces to develop its own report. The Cut tin,:: 
Edge , Xv'hich was transmitted to the governor and 1987 
Legislature. During this time the CHE had an interim staff 
director, and a Republican, Caroll Campbell, icas elected governor 
in November 1986. The 1987 Legislature wa? fac ^ with- ^ 
hold-the-line budget because of revenue short falls, and did not 
take any action on the study recommendations from the CHE. The 
timing for the CHE ' s comprehensive improvement program was still 
bad. 

In summary, an objective and comprehensive study of South 
Carolina's higher education system was made by external 
consultants. The report was presented at a time when 
implementation was too difficult. It faced the additional 
problems of a weak state economy and changes in the leadership 
of the Commission and its staff, a new governor, and a group of 
institutional leaders who had no ownership and few incentives to 
support the recommendations. As a result, eighteen months 
later, there is still little evidence that many of the 
consultants' recommendations will be implemented, but a final 
evaluation must allow more time to see what develops. Fred 
Sheheen was named executive director of the Commission in the 
spring of 1987, and is renewing efforts to get public, 
institutional, and legislative support for the recommendations of 
the Commission. 
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Summary 

A revieiN^ of these six studies emphasizes ine importar4ce of 
situational factors in determining their impact. Five of the 
studies produced reports dealing with a broad range of important 
policy questions. The sixth report, West Virginia, x^as addressed 
primarily to the controversial governance question and the 
functioning of the Board of Regents. 

The Commission in Washington, which took the longest amount 
of time and spent the most money, had an indirect influence cn 
the subsequent change in governance, but most observers thv:>ught 
the change would have occurred ar.-Kay. The Washington report also 
was ill-timed, being presented to « new^ governor with uncertain 
interests in the issues. It ua.s a well done study, but it had 
too many recommendations and was released at the wrong time. The 
ideas it developed about coordination were eventually reflected 
in the changes that were adopted by the legislature following the 
sunset review and the tx>70 legislative study committees. 

The South Carolina re pert had no impartial citizen group to 
enhance the credibility the CHE could give it. Its report uas 
issued at a time when the key leaders who had promoted or 
supported the study were about to go out of office. The 
economies of Washington and South Carolina were not expanding 
Xvhen the reports Xvere released, and there was little opportunity 
for new revenue to fund nex>7 initiatives. In fact, no new taxes 
were being enacted in any of the states studied. 

The Idaho study was sponsored by the business interests in 
the state and had no governmental authorization. It used outside 
consdltants to develop a series of recommendations. Better 
funding for higher education was a central recommendation, but 
the combination of a Xv'eak state economy and a conservative 
legislature led to failure of the tax initiative Xvhich could have 
provided better funding. The business leaders in the state x:ho 
had sponsored the study Xs'ound up opposing the tax increase. This 
study gets lox>7 marks for impact, although it raised the level of 
debate in Idaho on \igher education issues, and some of its ideas 
are reflected in other reports. The lack of a governmental 
sponsor for tLe study Xvas also a hindrance, because there x:asn't 
sufficient "ox>7nership" of the study by either the governor or 
legislators . 

Thp West Virginia consultant study may have increased the 
awareness of legislators and the executive staff about options 
for changing the governance of higher education in the state. 
The study also made suggestions about making the existing system 
more effective and some of these x>7ere adopted by the Board. 
Hoxvever, the report was commissioned by legislators uho had 
adversarial relations with the Board of Regents, and the report 
was not a very good platform for formative suggestions to the 
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Board. The West Virginia study was narrowly focused on 
governance issues; more important questions about quality of th? 
higher education system and finance were not examined. The study 
did not clarify these issues for the public agenda. 

The Florida Joint Executive and Legislative Commission had 
more direct impact. Several of the recommendations of th;* st\;dy 
commission were adopted. A major one was the establishment of 
the Postsecondary Education Planning Commission (PEPC) as a 
comprehensive planning and policy analysis body for the sta*::? 
Florida. PEPC has been a valuable addition to the governance 
arrangements in Florida. The commission's other recommendations 
about role and mission of the south Florida universities hav>:- 
taken longer to implement. In summary, the Joint Commissicn t^^ok 
up a set of specific idsues, had good staff work, and came- ca;t 
with recommendations which had fair success in implementation. 

The best record of implementation goes to the Colorado Study 
Commission on Higher Education. The study did a good job of 
reviewing the policy issues and needs of the state. It had only 
one legislator on it, but he was persuasive in introducing 
legislation which led to a restructuring of zhe CCHE and some 
strengthening of its responsibilities. In addition, the 
legislation provided an ambitious policy agenda Tor higher 
education. It remains to be seen what the long-run impact of the 
changes in the Commission will be for planing and policy-making 
in Colorado, but certainly the special Commission got high marks 
from nearly everyone. The Commission's report led directly to 
legislative action which restr ^tured the Commission and gave it 
a broad policy agenda. 

This review and analysis raises some questions about the 
Johnson and Marcus list of factors which make for an effect i\'e 
study. 

1. Having a long time to do a study does not necessarily 
have a positive effect on the impact of the study. Special 
studies are not research projects, but are policy studies that 
bring the best available information to bear on the issue. 

2. Outside consultants are problematic in their effect. 
They can be helpful in ensuring objectivity or in addressing 
technical problems, but some of the best staff work for 
commissions was done by regular state employees who were already 
familiar wi'h the background, the state operations, and the 
issues at ha-nd. 

3. Timing of studies is critical. Even a mediocre report 
that appears at the opportune time may have some imx-act while an 
excellent report at the wrong time is likely to go nowhere. When 
there is no new money to support the recommei-iations , the chances 




of action are reduced. When key supporting leadership goes out 
of office, the chances of action are reduced. 

4. Spe* ial studies seem to be more suitable for examining 
some issues than others. Studies that have complex technical 
aspects are not as easy to examine with a citizens study 
commission as are broad matters of policy . They can rode fine 
issues and help to develop a consensus on needed action. They 
can take a long-range viex>- rather than being focused on the 
immediate problems . 

All the stuJy commissions examined made recommendations of 
one sort or another about governance issues and about the 
functions of the statexvide agency. With the exception of th?* 
West Virginia study, each examined governance issues in a broad 
policy context. Governance and organizational issues aro issues 
which commissions are usually asked to examine. This is partly 
because state governance of higher education is a nex> policy area 
in which arrangements are not set by historical precedent. 
Constructive outcomes are more likely xvhen the governance issue 
is examined in the context of the broad policy objectives of the 
state, rather than being considered as a matter of state power 
and regulatory control over colleges and universities. 

5. The study commissions had little impact on fiscal issues, 
riscal policy is hard to change because so many groups are 

involved in making decisions. The governor, several legislative 
committees, and multiple interest groups are interested in every 
outcome , and the possibility of change of any sort is limited . 

6. Special studies also provide a nexv look at the opticns 
about an issue, and they may be able to redefine the issue in 
ways that enable a majority of people and interest groups to be 
"satisf iced'* and to arrive at a compromise solution. They aro 
not gocC for doing new studies and collecting neu evidence about 
issues or verifying analyses that challenge the conventional 
wisdom about a topic. They are usually not good for folloxs-up, 
or for any ongoing assessment process that is designed to 
stimulate continuing improvement . 

Special studies can perform tvo other functions that are 
hard to assess. First, they can help develop a consensus among 
state laaders about what the important issues are. This is a 
necessary prerequisite to effective action about any issue. 
Legislators and higher education leaders sometimes talk past 
each other because they define the issues in very different xcays. 
For example, one person^ s needless program duplication may be 
another person's healthy competition to prevent monopoly and 
complacency. 

Second, a study can sensitize the public to issues that need 
to be on the public agenda, and can broaden the consideration of 
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new options for dealing Kith the issues. By raising the level o 
public discussion of issues, studj' commissions can sometimes set 
in motion other processes that may lead to action in the future. 
As our case studies have shown, action on recommendations may 
come several years after the study, after considerable 
discussion and interest group bargaining. 

In summary, on the right issue at the right time special 
studies can be a catalyst for change and can have a bi^ impact. 
On other issues they seem to be less effective. Special studiiBs 
are not a substitute f or an ongoing evaluation process that 
provides f eedback for change and improvement . 
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Chapter VI 

COMPREHENSIVE EVALUATION AT THE STATE LEVEL 



Comprehensive assessment of higher education on a national 
or state level is a recent development. Chester Finn pointed <:ut 
in a Change Magarzine article in 1983 that we have almost nc 
systematic knowledge about what college graduates know, or ^'hat 
they can do with what they know* The initial plans for the 
National Assessment of Educational Progress in 1969 ^--er*^ to ha\ e 
a young adult sample, but it was dropped after one assessment 
because of budget cuts. There are a number of indicat:>rs about 
the quality of entrants to college, about the coherence or lack 
of coherence of the curriculum, and about the resources going 
into education. Some of these input measures have been 
monitored for decades, but the lack of a strong relationship 
between resource measures and outcomes in terms of student 
learning leaves very sketchy and impressionistic^ picture cf the 
effectiveness of our higher education system. 

As long as the effectiveness of higher education \:as not :n 
question, assessment of it was not a high priority. Polls takrii 
of students and recent graduates have revealed fairly high levels 
of customer satisfaction with the college experience; colloge 
graduates have generally been able to get jobs that made colle:^-? 
seem worthwhile. In the last decade the attention of Etat*- 
education leaders has shifted from concern about growth and 
access to concern about quality* Elementary and secondar:* 
education quality was the first target, with A Nation at his!: avA 
several other reports. The quality of undergraduate cell 3^'^i 
education was the ne>:t target ♦ Involvement in Learning 
was one of several national reports on the need for imprcvvmi^r«t 
in higher education* It recorr. lended more attention to aKsossr.on*: 
as a guide to improvement efforts. 

The national studies of undergraduate education got tho 
attention and interest of the governors who led state Gdu?aT:ior. 
reform movements. The Governors Association report Time For 
Re sults callev^. for comprehensive assessment of higher education 
to assure that institutional improvement efforts were working. 
In the 1980^s assessment has been pushed by state leaders, more 
than from wi thin the academy . 

A recent ECS survey (ECS 1987) of state assessment cclivity 
indicated that nearly all the states are either doing something, 
planning and developing something, or discussing some sort of 
comprehensive higher education assessment activities. As the ECS 
survey points out, some states like Florida, Georgia, N'ew Jersey, 
Tennessee, and South Dakota have explicit assessment programs 
that involve testing or other measures of educational outcomes. 
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Some states have built institutional and program assessment 
into thoir planning, budgeting, and mission review functions. 
Other states resist treating assessment separately; thor ccnsid-^r 
it more appropriate to conduct assessment within a broader 
framework of existing policy mechanisms such as statewide maatoj 
planning, mission approval, or program review ... Ohio^s respon.^c 
is typical of this group . . . "We are not focussed on a?,se'ismcn* 
per se , but are dedicated to improving the quality of higher 
education in Ohio. To get a qualitative improvement, ^'c- !^•il3 
naturally get an assessment *by-produc t ' " (ECS, 1937, p. 2). 

The kind of role states have defined for themselves in 
higher education planning and coordination will shape th.e 
approach they take to assessment. Et^^ell ( 1986 , p. 50-^1 ) 
identifies three different approaches that states have taken. 
One sees state assessment as an act of quality assurance?, a 
second sees assessment" . . . primarily as a way of ensuring tha* 
institutions are making improvements along a number of 
performance dimensions deemed important by the state." The final 
approach sees assessment primarily as a device to promote 
institutional self-improvement. The differences in conductlnir 
assessment for improvement versus assessment for accouT*tabi 1 L ty 
are illustrated by these different approaches. 

If states take the quality assurance approach, they will 
seeK common measures or indicators of institutional quality. 
Using elementary and secondary assessment as a model will Icv^i 
the state to emphasize testing of all students in all 
institutions, and they tend to assess them on a narrow rang*; <: f 
knowledge and skills. Florida's College Level .-^cademio Skill:- 
Test (CLAST) program is an example. It measures knot;ledgc in 
reading, writing, and mathematics. Very few states are using th*^ 
common test approach; the diversity of programs and inst 5 tut icn?- 
makes uniform testing an inappropriate procedure. 

When the state takes the second approach to measure 
achievement of a broader set of state objectives that ar»? r-^l-i' -d 
to quality improvement, multiple assessment procedures ar»j 
needed. The objectives often include general education 
achievement, achievement in the major field, and emplcy:neni 
records of graduates. Tennessee has a performance assossm-^nt 
program which has five different measures of institutional 
performance. Two of these are student achievement measures; th? 
others include accreditation, student and employer satisfaction, 
and planning for academic improvement. Incentives for improvin;^ 
scores on each indicator are built into the budget process. Xe^" 
Jersey is developing multiple assessments of general education, 
the undergraduate major, and other goals; although they are not 
linking assessment results directly to the budget. 

The third approach of encouraging more institutional 
self-evaluation institutions flexibility in determining 
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their OKn assess^nent procedures. The state requires that the 
institutions initiate some assessment process, and report on 
results to the state. States with th.s definition of the state 
role in assessment are operating in a similar mode to the 
accrediting process. The new demands for institutional 
assessment for this third r^oup of states may not be :^iuch 
different than the institutions have been involved with in the 
past . 

A state outcomes assessment process involves the sensitive 
area of state-institutional academic relationships. Institutions 
have claimed the freedom to pursue their educational objecti\e3 
without an} political or bureaucratic intrusions from governnient. 
This freedom is important to the univt^rsi ty ' s role in advancing 
knowledge and in teaching without state censorship. If the 
state is involved in assessing academic performance of 
institutions, it is only a small step for the state to get 
involved in mandating changes in order to remedy weaknesses 
discovered in the assessment. Institutions fear that this might 
get -Lhe state to mandate curriculum changes or involve itself in 
inappropriate ways in the teaching and learning processes of 
higlier education . 

Even if the state observes what Clark Kerr has called the 
"self-denying ordinance" and refrains from any intrusion into the 
management of the institution, assessment results Kill oecome 
public knowledge, and unfavorable, results can give the 
institution a negative image, hurting its student recruitment, 
its private fund raising, and possibly its appropriations. Evan 
when the assessment is done within the framework of institutional 
self-improvement, the possibility of a summative use of forms.tive 
assessment results can make institutions very uneasy about any 
state role in assessment. 

State higher education agencies that are involved in 
implementing assessments can get caught in the middle between the 
state's expectations for accountability and the institution's 
expectation thac it should set its own educational standards and 
procedures . 

A multi-year effort to develop a comprehensive evaluation 
program takes time to develop because it is difficult to agree on 
statewide objectives, develop acceptable measures of those 
objectives, and decide how they will be differentially assessed 
in different kinds of institutions. In addition, assessment 
approaches will be required in vocational as compared with 
academic programs, and in different fields of study such as 
chemistry, business, and social work. A lead time of throe or 
four years for design, development and implementation of a 
program of this complexity has been typical of the states like 
Florida, Tennessee and New Jersey that have been pioneering 
comprehensive programs. Even when a state can benefit from the 
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experience of another state's prior Kork, it must adapt borroxv-ed 
procedures to fit its own goals, and a minimum of two or three 
years will probably be needed to introduce new comprehensive 
programs in any state • 

Therefore, a review of comprehensive evaluation procedures 
should be possible when new programs have been implemented In 
enough states in different ways that provide a basis for 
comparison of their eff ectiveness . 

The effectiveness of comprehensive assessment procedures in 
stimulating and guiding improvement depends on whether the 
process makes a difference at the department and classrocr: iav^l. 
Unless assessment affects the teaching and learning process, it 
won't be measurable in the student outcomes • 

This review of state evaluation in the seven states did not 
include a detailed examination of the new comprehensive 
evaluation procedures. Florida is the only one of the states we 
visited that was implementing a comprehensive process in 1985. 
The other six states are still in the discussion and development 
stages. Thifre are about ten states, including Florida, where the 
state is assuming an active role in developing a comprehensive 
assessment program with multiple measures of institutional 
outcomes. Florida has been developing assessment procedures for 
nearly ten years. A preliminary review of Florida's 
comprehensive assessment process follows; a more definitive study 
of comprehensive assessments is planned for a later study. 

ComiPrehensive Assessment in Florida 

Florida has been developing comprehensive approaches to 
planning and decision making for more than two decades. Because 
the state was growing rapidly, the need for planning was clearer 
than in many other states. The state developed an early plan for 
community colleges which led to a comprehensive statewide 
development. Florida was and is a national leader in community 
colleges x^hich are designed as the primarj^ access to higher 
education in that state. Florida also added six new public 
universities to the three that the state had at the end of World 
War II. The new universities were planned to complement the 
community college system, and the 2+2 concept made four of the 
six new universities only upper division and graduate. 

Florida is also ambitious to have a high quality- system that 
will enhance the economic development of the state. A model for 
Florida has been the University of California with its nine 
campuses. Florida also had nine campuses in its state system, 
and aspired to have one or more of them recognized as peers of 
U.C. Berkeley or UCLA. 
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Another factor that led to the development of assessment 
xcere questions on the part of some legislators about the basic • 
skills ability of undergraduates in the Florida system. 
Legislative concern about basic skills arose s'^veral years before 
attention to quality became a national issje. It led to the 
development of the College Level Academic Slrills Test (CLAST), an 
exam given at the end of the sophomore year that tests basic 
academic skills. Students have to pass the exam to be admitted # 
to the junior year in any of the public universities. 

Education leaders also developed the goal of being in the 
top quartile nationally in education, including higher education, 
and promoted this goal to the governor and legislature. In order 
to determine where Florida education x>'as in relation to the ton # 
quartile goal, Florida began in 1981 to develop a set of 
indicators of quality by which elementary/secondary educa.ticn, 
community college';, and universities could be a.ssessed. After 
r.^ore than a year of development a set of 21 quality indicators 
was developed for the universities. They included four output 
measures (for example; test results, follow-up studies of # 
graduates) nine input measures (dollars per student, admissions 
test results, earned degrees of faculty, etc.), seven process 
measures (achieving specialized accreditation, achievement of 
equal access goals, etc.), and one opinion measure (national rank 
of s(ihool or program on peer ratings). 

These indicators are assessed each year and an annual rep:>rt 
is made to the governor and legislature. There are problems T:ith 
developing v^uality indicators. Information in comparable form 
for output measures is not available for other states, so Florida 
can only compare itself on input and on some process measurers to 
see if it is in the top quartile of states . % 

When comparative measures for other states are not 
available, Florida can compare each institution's progress with 
its past record, or with other in-state institutions that have 
similar programs and objectives. For example, institutions can 
be compared on "uhe achievement of general education objectives, % 
but these comparisons have to be adjusted for the ability level 
of the entering students. Diversity of institutional purposes, 
programs, and students make all of these comparisons difficult 
and subject to misinterpretation. 

Some of the indicators, such as Graduate Record Exam scores, ^ 
are very general, and give an institution little guidance about 
the programs or disciplines that need to improve performance . As 
a guide to institutional self-improvement they are of limited 
value . 

The impetus for the indicators of progress came from the ft 
State Boaru of Education and the legislature. The Commissioner 
of Education was committed to develop indicators of progress for 
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elementary/secondary education. He insisted that higher 
education develop a similar set and that they include output as 
well as input and process indicators. Since in the Florida 
system the Boara of Regents for the nine universities is 
ultimately responsible to the State Board of Education their 
participation in the development of the indicators was mandatory. 

One motivation for institutions to participate in the 
indicator project is the hope that improvement of the indicators 
will cause the legislature to take a more positive attitude and 
give higher education more money. Since the indicators are not 
linked directly to budget allocations in any way, the effect cf 
indicator changes on appropriations to higher education is 
indirect and problematic. The evidence is that higher education 
has not changed its percentage share of the state budget up or 
down by very much in recent years. Under these conditions wo 
should expect institutions to regard indicators as bureaucratic 
requirements (which they are) more than as guides to improvement 
or change. 

The Florida Board of Regents also developed a program revie*. 
process more than a decade ago. All programs in the nine 
universities are assessed on a periodic cycle. The reviews by 
outside consultants can lead to program modification, increased 
resources for program improvement and in selected cases, pro^rair. 
termination. This process is linked to .the planning cj'cle, and 
to ills ti tutional role and mission determination* It has had a 
direct effect on resources and on institutional actions, and is a 
type of evaluation xchich can be useful both for institutional 
self-inprovement and for state-level resource allocation and 
planning decisions . 

The Florida legislature wants to see planning, budgetin-5 and 
evaluation processes linked at che institutional and state level. 
The Postsecondary Education Planning Commission develops a 
statewide plan that includes all sectors of pos t secondary' 
education, universities, private institutions, community colleges 
and vocational education. Planning objectives are used in the 
budget development and review process by the Governor and the 
legislature. Evaluative data from the CLAST program, from the 
indicators project, from the statewide program reviews can all 
be used as input for budget decision making. Institutions take 
these assessments seriously, but the jury is still out on their 
impact on system improvement. A future review will be able to 
examine these issues in more detail, both from the state 
perspective and f rom the perspective of the ins ti tut ions , 
faculty, and students. 

Respondents in Florida gave these different assessment 
activities a mixed review. While they generally accepted the 
state role in comprehensive assessment as legitimate, several 
persons were quick to point out that indicators had real 
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potential for misinterpretation, and gave at best a very limited 
picture of higher education accomplishments. The CLAST exam also 
got a critical appraisal. Some college people thought that 
measurement of basic skills proficiencies should be moved back to 
the end of high school. The Florida program review process got a 
more favorable assessment by higher education respondents . 

One of the major institutional criticisms of Florida's 
assessment activities was that they had not led to increased 
funding. While total dollars have groxvn, the constant percentage 
share of total state revenues going to higher education was 
evidence to our respondents that the size of the budget is cnZy 
marginally responsive to higher education needs, as revealed in 
^.he assessments. 



Summary 

Comprehensive state assessment of higher education will 
likely be initiated in more states than the eight to ten that are 
already involved. The particular forms that it takes-- 
indicators, tests, program reviews, or multiple assessments-- 
will evolve during the next fex>? years with states adopting 
dif f erent approaches . 

The key issue with comprehensive state assessment procedures 
is xvhether or not they will lead to improvement of the education 
of students and the research and service functions of the 
institutions. Comprehensive assessment of outcomes and 
effectiveness is promoted by governors and legislators ; the 
impetus is primarily external to the academy. Assessment data 
will be used for summative evaluations; they may affect 
institutions' budgets, programs, and overall role and mission. 
Whether they will also be used for formative purposes of 
institutional self-improvement is difficult to judge at this 
early stage of their implementation. Institutions have enough 
program and substantive autonomy that they can probably meet most 
of the state goalsv by administrative adjustments Xv-i thout actually 
changing how students are taught or what they are taught. The 
possibility of playing ''assessment games" that are similar tc 
"budget games" is alxvays present. The assessment game is to be 
able to report progress, whether it occurs or not. It may be 
easier to find xvays to report improvement, than to engage in the 
more difficult task of making improvements. 

The states xvhere each institution is free to develop its oxen 
assessment procedures have a different problem. Institutions 
xvill develop their oxvn measures of performance in relation to 
their own goals. It Xvon't be possible to combine the varied 
institutional assessments into a statewide picture, and it Xv^ill 
still be difficult to answer the question: "What do our college 
graduates know; ho\^ well is the state doing in higher education?*' 
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Institutions that are motivated to improve themselves will 
develop a comprehensive assessment program, ichile institution-^ 
that are satisfied with their current situation will do as little 
as possible to meet state requirements, since they have no 
internal motivation to do more. 

Regardless of the approach the 3tate takes to evaluation, 
the State Higher Education Agency should be the leadership group 
in the evaluation process. To the extent that evaluations are 
used in state decision making and policy formation , a.nd in 
shaping public perceptions of higher education, the complexity of 
the shea's job will be increased. In view of the political and 
economic influences illustrated in the states studied for tMs 
report, the success of SHEA evaluation activities Kill likely be 
determined by the agency's ability to integrate assessment data 
into planning and budgeting cycles. 

Assessment processes will test the skills of state higher 
education executives who must be persuasive in presenting 
evaluation data to state officials, the media, and the 
legislature to reduce the chances of misinterpretation. The 
executive will also have to deal with the fairness, scope, and 
accuracy of any assessment process and results. It will take a 
diplomat's abilities to deal with state officials' desires for 
more assessment and with institutions that are sensitive about 
assessment information. College administrators will want to 
control any evaluation process that applies to their institutions 
while the governor and legislature are likely to want the 
procedure to be objective and independent, on the audit model. 

At this stage it is too early to make a judgment about the 
impact of different approaches to state assessment of higher 
education. Will the decentralized approach lead to more 
institutional change and improvement than the more centralized 
efforts of states like Florida? Will the decentralized 
approaches be so diverse that they give no guidance to state 
planning and resource allocation? These questions will be 
examined in more depth in a later study. 
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Chapter VII 
CONCLUi:iONS AND RECOMMENDATIONS 



Higher education systems and state higher education igencies 
are evaluated in a variety of vays . This report reviewed several 
types of evaluations in a survey of seven states , as siammari zed 
in Table One of Chapter I. While these seven states were not a 
statistically representative sample of the fifty states, they 
included examples of most of the evaluation activities currently 
being used to assess policies and programs, 
conclusions and recommendations undoubtedly 
opinions and biases as well rs the comments 
state observers . 



The following 
reflect the authors* 
of respondents and 



Different forms of evaluation have been applied at different 
levels of higher education systems. More importantly, to conduct 
narrow or unilateral evaluations without adequate consideration 
of many variables is often of limited value. To ensure that 
value is derived from the process, the various forms and levels 
of evaluation need to be brought together through careful design 

achieve that good fit is a complex issue 
The recommendations given here may help 
and urgency ot making the correct 
connections between evaluation approaches. In Chapter I, the 
four main conclusions were previewed in ascending order of 
importance and descending order of certainty. These are now 
given with justification derived from the intervening chapters. 



and cooperation. How to 
demanding further study, 
establish the importance 



SHEA Self-Study Evaluation 

Formal periodic self -studies by state higher education 
agencies are not common. While virtually all colleges and 
universities use outside consultants to help them reexamine thei: 
purposes and programs on periodic cycles that are established by 
accrediting agencies, only Alabama among the fifty states has a 
similar process for its state board. A number of other SHEA'f 
have had informal retreats to examine their own functioning, and 
some states have had more formal reviews with external 
consultants at critical periods in the agency's history. Given 
the state agency's important role in comprehensive planning and 
systematic program review, Lt may seem paradoxical that they do 
not apply the benefits of periodic self-assessment to their own 
operat ions . 

One reason for the failure of most SHEARS to adopt periodic 
evaluation procedures is the newness of most of the coordinating 
agencies . The need for state planning and regulation of higher 
education has been recognized in most states only in the last 
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twenty-five years. The role and authority of these new agencies 
has been challenged in most states by either the institutions or 
the legislature or bothl In a majority of states, there have 
been changes in the role or authority during the last ten or 
fifteen years sometimes accompanied by changes in personnel. In 
the unsettled political environment that exists for most SHEA's, 
evaluation can be seen as more of a threat than a help. 

A second reason is that the SHEA is a '^boundary" agency, one 
that works between the governor, legislature, and higher 
educatior institutions. The effectiveness of the SHE.A is Judged 
in the final analysis by the way the whole higher education 
system functions, and x>hether the state adopts policies which 
enhance the opportunities for citizens and the quality of * ..eir 
education. The SHEA is not fully in control of the system (ever, 
if it is a state governing board) but is dependent on actions of 
the legislature , institutional leaders , and the governor to 
formally adopt and implement state policy goals. A 
self-evaluation that looks merely at the inner workings of the 
SHEA does not help much in assessing the role of the SHEA in the 
effective performance of the higher education system in the 
state. 

It is our belief that most of the SHE.A' s are mature enough 
to initiate a periodic self-evaluation process, and that the 
reyiew of their activities by outside consultants as a part of 
the self-evaluation is likely to be more helpful than simply 
having a staff and board review of their own operations. 

The self -evaluation needs to assess the way the SHE.\ 
performs its policy role in improving the state system of higher 
education and its policy role in assuring that the citizens hav:> 
adequate higher education opportunities. This will involve mora 
than the internal Xvorkings of the SHEA although these should 
certainly be examined. Assessment of the broader goals involves 
looking at the way the whole system works, and the role of the 
SHEA in the pol'cy process. When the SHEA is examined in 
relation to the institutions, legislators, and the executive 
staff of the governor, it moves the process beyond 
self -evaluation , and opens it up to summative as well as 
formative judgments . If the state agencj" has an adversarial 
relationship with a major university or system of institutions, 
for example, the self -evaluation may open SHEA to criticism that 
damages its credibility. But the potential benefit to the state 
of subjecting the state policy process to scrutiny- should 
outweigh the problems that such reviexv' might create for the 



The SHEA should also consider the value to its long-run 
survival and improvement that might folloxv from it undertaking a 
self-evaluation and thereby having developed the first draft of 
criteria cf evaluation which later summative processes might 
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incorporate or modify. It may not later be as threatening if the 
agency has made a pre-emptive effort. 

One xvay to deal with the self-evaluation of the SHEA in the 
broader context would be to incorporate it into the plannins 
process. The periodic review and update of the state's master 
plan could include a self-evaluation of the SHEA's role in 
de^ling xvith the major policy issues that were included in the 
plan. This could help < et the proper context for the evaluation 
which should look for the best way the state's higher education 
policy objectives can be advanced. 

The self-evaluation ought to be set up on a regular cycle 
that alloxvs the state system and the SHEA to be evaluated in 
"normal" circumstances. The more common incremental approach is 
to put off evaluation until a crisis arises that requires some 
c^ction. Evaluations done in a crisis inevitably focus on the 
immediate solution of the particular crisis, and therefore 
overlook the broader issues and the long-term interests of the 
state in favor of an immediate answer to a problem. They 
sometimes move prematurely toward an agenda of restructuring 
and/or shuffling key personnel. Furthermore, reforms proposed in 
a context of crisis may be evaluated against the short-run 
criterion of xvhat they do to "solve" the problem. 

Our first recommendation is that SHEA's should initi^-^^ 
periodic self-evaluation of the effectiveness of their a^pr^- in 
dealing with the ma.ior policy issues in the state. Thero are 
several possible patterns for these reviews, but the use of 
objective external consultants is likely to make the reviews more 
useful as xvell as more credible. 



State Performance Evaluation of the SHEA 

A number of states have done performance audits or sunset 
reviews of the SHEA. Since over forty of the states now have 
performance audit programs and staffs, the potential exists for 
most of the states to do an audit of their SHEA. The ^performance 
audit process appeals to executive and legislative staff because 
it is objective . A self -evaluation can be self-serving ; the 
rudit process , by contrast , conforms to the accountability 
requirements that states are insisting that all public agencies 
meet. 

Our review of performance audits indicated that they had 
major limitations as procedures f.-r improving the state policy 
process in higher education . 

Their first limitation was that they usually examined the 
SHEA narrowly without examining the broader state system in Xshicb 
it was functioning. The state agency could be doing things right 
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without doing the right things, and the performance audit is mor 
likely to use the legal responsibilities of the SHEA as the 
criteria for evaluation than the pol5oy objectives of the state. 

The second limitation was that performance audits, 
especially sunset reviews, tended to focus more on whether the 
agency should be continued in its present form or not, than on 
the question of how the state- level functions of planning , 
resource allocation and evaluation could best be performed. It 
is harder to assess these broader issues, but they ave the 
important quest i ons . 

A third criticism of performance audits is that they may be 
made by persons who do not have adequate knowledge of higher 
education to make complex and sensitive judgments about it. Thi 
assumes that performance audits should be based on peer judgment 
and in the absence of clear and objective criteria, the 
performance audit will be largely judgmental. Our review found 
that some performance audits were based on a considerable effort 
to develop appropriate criteria and measures of performance, and 
others were primarily judgmental. The fear of insensitive anl 
uninformed auditors was not justified in most of the audits we 
reviewed." 

A final criticism is that performance audi"cs were often 
politically motivated effoits to criticize an unpopular agency d 
tc do an audit o"^ an ineffective agency to improve the image c f 
the audit process with the legislative oversight committee 
members. In such a process, even-handedness may be lost; 
criticisms are more welcome than praise. If the audit reveals 
weak performance, the justification for the audit process (and 
the audit staff) is strengthened. The political nature of ichD 
is selected to be audited x^as evident in several of the stat'?£ 
visited, but this is more a criticism of the legislators who 
make these determinations than it is of the audit staff. Any 
evaluation process can be used for political purposes. 

Our second recommendation to states is that thp p<=^r f or-^anr-*:^ 
audit process is inadequate as an evaluation of SHEA's 
performance . Even when the evaluation is done well, it does not 
deal with the key issues that need to be examined. Our 
recommenda ti on is not that the state agency avoid accountabil i ty 
for effective performance, but that another process be used to 
evaluate the SHEA's in the context of the functioning of the 
entire state system. The remaining recommendations suggest some 
structure for more productive evaluations. 

Special Studies and Blue Ribbon Commissions 

The most common form of evaluation of higher education has 
been the special study, done by a consultant group, a group of 
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distinguished citizens, a legislative committee, or by some 
combination of these. Even after development of statewide 
planning agencies in the sixties and seventies, special studies 
continued to be done, and they ^an be a useful form of 
evaluation, supplementing the studies and policv recommenda t ^ on-s 
of the SHEA'S. 

Common situations in which special studies may be 
appropriate are: 

1. When state authorities Kant an objective overview of the 
state system of higher education and the x>?ay the SHEA functions, 
they can appoint a citizens committee and/or employ outside 
expertise -"-o do a review, California has a periodic re\iew" of 
its master plan approximately once each decade, using a citizt>n 
group and a special staff and consultants to examine the major 
policies guiding higher education. The California Master Plan 
Review recommends changes that align the state's higher education 
resources with the needs of the next decade. Other states ha\ e 
also used special commissions to look at the present performance 
and future directions of their higher education system, althoui?k 
they usually do this on an ad hoc rather than periodic basis. 

2. When a broad poj icy issue or issues arise that 
involve(s) the jurisdiction and efforts of several different 
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state groups, a special study group may be needed. Heal 
professions education -xnd teacher education are two examples 
where the SHEA will need to work with the State Health Departrr. :^nt 
and State Education Department, respectively, in examining and 
developing recommendations . 

3. If the issue of state-level governance structur^^ n '^i-i 
CO be examined, an impartial outside group will give more 
credibility and objectivity to the analysis than anything tho 
SHEA might recommend about itself. 

The work of special studies and blue ribbon commissions is 
often judged too narrowly on the extent to which their 
recommendations are subsequently adopted by the legislature. 
There are other criteria by which these studies can and should bo 
evaluated. One is whether they successfully raise relevant 
issues for discussion, and whether their x^ork contributes to the 
education ol' the public, the legislature, and other key 
policymakers about the nature of the issue and the strengths and 
weaknesses of alternative solutions. They also keep important 
problems on the public agenda so that the legislature and 
governor will deal with them. A third criterion for avaluatJng 
the work of commissions is their ability to bring together the 
key groups that can bring about a resolution of an issue. 

When looked at against the first criterion, whether the 
special :'tudy's recommendations are subsequently adopted by the 
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legislature, the studies ue reviewed had a mixed record. The 
best chance for legislative adopcion probably occurs when a 
legislative committee or legislatively appointed committee doz'S 
the study. Since legislators are partisan, the legislative 
committee may use the process to reach a short- run poll t leal 
accommodation or compromise, rather than examining the issuts iT:-?r*? 
objectively and developing recommendations that reflect th3 bast 
long-term educational solution to the problem. When tho 
legislature and governor jointly appoint a distinguished lay 
citizen committee, as was the case in the Florida and Colorad:: 
study commissions, the chance for legislative action is ?ocd. A 
group of distinguished citizens can bring a nonpartisan and mor*^ 
objective look to the issue than is likely with a study c^roup 
made up primarily of legislators. 

When examined in relation to the importance of the issuoH 
they raised and the contributions to the education of key policy 
makers about these issues, the blue ribbon commissions we locked 
at also have a mixed record. Consultants can be useful in 
raising the issues in a comprehensive way. A blue ribbon citlzon 
group can be more effective in getting issues on th*'^ political 
agenda \.han a consultant group. Sometimes the blue ribbon 
committee can even have a second life after it makes its formal 
r port which gives them time to push for adoption and 
implementation of recommendations. The combination of 
influential citizen members and outside consultants has inovo 
potential strength than either group by itself. 

In summarv, special study commissions can play a useful r:»l'> 
in some situations and will probably- be needed on special 
occasions even in states where the SHEA is an effectivtii 
policy-oriented agency. In particular, we recommend the neriodi ^ 
use of a citizen committee and expert consultants to t^vnln^f^ ^ 
make sug.s^est ions for improvement in the state hi<^her pduo^^^in- 
system, its policies, s^overnancej and operation . If that r^^-i.'-w 
is part of the state planning process, as the California master 
plan review is, it can accomplish an i *portant dual purpose cn a 
pe-riodic basis, probably every ten years or so. We think thi.^ 
would be a better approach to accountability than the 
performance audit or sunset review, but it should be conducted as 
a periodic review - a step-bj.ok look over a fairly long period ~ 
as opposed to a reflex aotxon to a real or perceived crisis. It 
is the relationship of special commissi ons to other evaluat ions 
and to planning that will make the blue ribbon report most 
relevant and likely to be implemented. 

Comprehensive State Planning and Evaluation 

The recommendations we have made, including the value of 
p'i ")dic self-evaluation, the problems with performance audit, 
and the contribution of special study commissions, are based 
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primarily on our analysis of existing data and on professional 
judgment. Such evaluations can make a valuable contribution to 
the policy debate and decision making process, but they also 
have limitations which have been reviewed above. States have 
begun to look for accurate information on the outcomes of 
education and better information on the "results" the state is 
getting for its investment in higher education. 

Therefore, states have begun to require more systematic 
on-going measurement and assessment in education to give evidence 
about the quality of education. States also want to assess 
whether reforms are bringing improvement to the system. Those 
states that have been the most interested and active in the 
reform of their educational systems have also been the states 
that have been most active in developing comprehensi\ e as^jessnient 
ac tivi ties . 

It takes several years to develop a comprehensive evaluation 
system for higher edunation and in most states these activities 
are either still under discussion or are in the developmental 
stage. Because most of the states we visited were still in tho 
early development stages of their new state-level assessments, we 
did not examine this type of evaluation in depth at this time. 
Only Florida among the states we visited has a history of 
development of statewide tests ^ indicators of performance, 
state-directed program review, and other aspects of a 
comprehensive evaluation system. Florida's experience should 
prove useful to ether states as they develop assessment programs. 
We propose to visit several states in the future, when they will 
have had time to implement some of their nexv procedures. The 
state role in comprehensive assessment varies from leaving 1.he 
initiative with campuses and monitoring their procedures, to a 
very active role in setting indicators of progress, reviewing 
programs, and mandating statexvide testing. Florida, Tennesseo, 
Nexv Jersey, and Missouri are examples of states that are 
developing comprehensive assessment systems, but tl.ey are not 
alike in \^ha.t they are trying. 

Useful information about program and institutional 
performance cannot be obtained very xvell xs'ithout periodic 
systematic data collection. Ad hoc studies and commissions, and 
performance audits have to xvork xvith the available data, and can 
make limited contributions to the on-going planning, resource 
allocation, and management decisions xvhich must be made to gel 
sustained improvement in education « Systematic evaluation car 
provide performance information that will be most valuable at the 
institutional level, but some of that information is also 
important to state-level policy-making and resource allocation. 
The recent demands for micro-level assessment have generated 
concern about the use of data to shape budgets and missions, but 
successful connection of micro-level studies to ongoing macro- 
level planning and evaluation may prove to have the potential to 
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bring a sense of order, utility and sensitivity to evaluation 
procedures at all levels. While the states are getting mor*^ 
involved in comprehensi\ e asses£,ment programs, there are a numb-^r 
of problems that nf d exploration before this type of evaluation 
can fulfill its potential. 

The development of comprehensive ;issessments can have both 
beneficial and harmful effects on higher education. Good 
information about higher education performance and program 
effectiveness has the potential to improve state policies and 
resource allocations, and it also has a potential for misuse. 
The potential for improvement depends a great deal on the Kay *:he 
nex^ state role in this sensitive area is developed and 
implemented, and x>7ho has the responsibility for those fun?tion^. 

We recommend that the State Higher Education A^enry \-^k^ i 
stroni^ policy leadership role in helping thf^ instituMone nnf^ 
state officials to x^ork toi^ether to develop an effective 
assessment process . Activities relating to student assessment at 
the campus level and institutional assessment at the state le^'el 
should develop in coherent ways x^rhich are compatible xcith broad'^r 
evaluation processes. The SHEA will have the complicated job of 
linking fragmented assessment activities xvith xchatever macro- 
level statewide efforts are taking place. The institutions, x'ith 
their individual perspectives on state issues, cannot do this; 
other state offices lack the necessary sensitivity to acaderair* 
issues. Thus, xvhether taking the leadership in assessment or 
coordinating the separate institutional assessment activities, 
most shea's x^7\11 occupy a key position. If there are risks in 
moving too fast in this emerging area, there are also risks in 
moving too slowly; and SHEARS should sense their responsibility 
to use both their brakes and their gas pedals to help their 5l\itt- 
arrive at a set of policies balancea to the needs ana conditions 
of the state. 



We earlier noted that the four main recommendations xsere 
made in ascending order of importance and in descending order of 
our certainty. We now can explain those terms more fully. 

The easiest recommendation to make was one xvhich x%e offered 
with great certainty that some form of state board self- 
evaluation would be beieficial- But ue also recognised the 
limited importance and application of this action in the total 
range of evaluation activities . 

Somev:hat more visible and important as an evaluation process 
x^;ould be the various forms of state performance audits or sun.set 
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reviews. We felt quite certain that our conclusions regarding 
their limited effectiveness were correct. 

Even more importance and atte, ion is given to the many 
state studies using bl ue ribbon committees and/or out-of-state 
consultants, but our mixed assessment of the effectiveness of 
these patterns compelled us to offer them with less dogmatisni 
than the previous txvO conclusions. 

Finally, we noted explicitly the potentially most important 
process — the building of a comprehensive state assessment system 
and the leadership role of the state board in that process. Ke 
had not done substantial empirical field work on that topic, and 
the stat es attempting comprehensive evaluations are not readv f c r 
critical review. Therefore, we must of "er that recommendation in 
a more speculative mode. 

Others can react to these four recommenaa tions to let us 
know where we have seen things correctly^ and where we have 
erred ! 
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Appendix A 



ECS Surveys of Blue Ribbon Commissions and State Assessment 
Activities in Higher Education, 



Boyer, C, M, , Ewell, P, T,, Finney, J, 



and Mingle, J. R, 



(1987), Assessment and Outcomes Measurement, A View from 
the States: Highlights of a nevv ECS Survey and Individual 
State Profiles (IPS-87-1), Denver, CO: Education 
Commission of the States, 



This report includes findings from a 50-state survey conducted by 
mail and phone from January-February 198'^, The findings Eho'c 
that two-thirds of the states have ordered assessment activiiiies. 
State initiatives were categori zed in the report as follows : 
mandated statewide testing programs, early intervention programs, 
encouraging institutional action, assessment within existing 
statewide mechanisms, and statewide monitoring of other outcome^ . 
State profiles follow the narrative with contact persons listed. 

McGuinness, A, C, Jr, (March, 1986), Major State Hi°'her 

Education Studies 1980 to present , Denver, CO: Education 
Commission of the Stcvtes, 

Tlxis listing includes major studies that emphasize all or most of 
the state's higher education system. Each stai-e is included with 
the exception of Wyoming which does not have a large system of 
higher education. The studies are listed by type and include 
state board/agency statewide or master plans, state board/agency 
periodic or continuous planning, legislative committee studies, 
study commission reports, and state board/agency consultant 
reports , 



Tolliver, S. (March, 1986). 

Higher Education , Denver, CO : 
States , 



Summaries of States: Studies of 

Educa"cion Commission of the 



This report summarizes studies conduc ced in nineteen states : 
Arkansas , Colorado, Connecticut , Georgia, Idaho, Illinois , 
Kentucky, Maine, Michigan, Minnesota, Mississippi, Nebraska, Kew 
Jersey, New York, Oklahoma, South Carolina, Texas, Washington, 
and West Virginia, Information included in each state summary 
includes the membership/staff, the date, the issues covered, the 
recommendations of the study, and the current status. 
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Appendix B 

Technical Advisory Committee Members ^ 

Patrick Callan 

Associate Executive Director 
Education Commission of the Stages 

1860 Lincoln Street, Suite 300 ^ 
Denver, CO 80295 

James Furman 
Vice President 

MacArthur Foundation ^ 
140 South Dearborn Street 
Chicago, IL 60603 

James Mingle 

Executive Director ^ 
State Higher Education 

Executive Officers Asisociation 
1860 Lincoln Street, Suite 310 
Denver, CO 80295 

# 

Lawrence Pettit 
Chancellor 

Southern Illinois University 
Colyer Hall 
Carbondale, IL 62901 

Sheila Tolliver 

Former E.cecutive ssistant to the 

Governor for Edu^^ation 
367 Dex^ey Drive 

Annapolis, MD 21401 ^ 
Aims McGuinness 

Education Commission of the States 
1860 Lincoln Street, Suite 300 

Denver, CO 80295 ^ 
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